Lesson: Face Detection with OpenCV

OpenCV is a library of image and video processing functions freely available for anyone to use. It is fair to call it an industry standard. OpenCV contains many capabilities accessible at various levels of complexity, but by and large, it is complex and difficult to learn and use. To make it easier to use and more powerful, there are classes in OpenCV that add higher level functions over the base OpenCV API. All of this can make learning OpenCV daunting task. However there are many tutorials and examples on the internet.

The Tello-SDK also provides a simple way to get started with OpenCV. It supports human face detection with it's FaceDetection class. FaceDetection uses OpenCV image object detection functions to find faces in images. Using FaceDetection is a pretty easy way to get started using OpenCV.

This exploration of face detection is intended to show what can be done with the Tello camera, which is really the Tello's only user programmable sensor. We can use OpenCV to process images from the Tello video stream and program the drone to respond to what it sees autonomously.

Take a look at the FindFace example program. It is a modification of the FlyController program. You can see in the program main loop there is a block of code that is turned on or off with the X button. This block of code makes a call to the FaceDetection.detectFaces() method. This method in turn makes the OpenCV calls needed to do a face detection process on the current image from the drone video feed. The method returns true if one or more faces are detected.

If faces are detected, the code requests a count of the faces detected and then a list of Rectangle objects that define the location and size of the detected faces in the image. Finally it uses a feature of the TelloCamera class that allows the programmer to provide a list of Rectangles that will be drawn on the current image of the video feed until reset. This will draw boxes around the faces in the live video feed and any picture or recorded video.

Modify the Main class to execute the FindFace class. Without taking off you can test the program with the drone in your hand. You can then fly the drone and (carefully) fly it to point at a person and test the face detection.

The comments in the code will tell you more about what is going on and as always, you are encouraged to look at the FaceDetection class code to see what is happening at the next lower level.

To learn more about OpenCV and image processing, view this basic tutorial. Here is more detailed OpenCV documentation and here is the JavaDoc.

One thing you will notice is that the drone flight controls and buttons are not as responsive as before. This is due to the fact that the face detection processing is done in-line in the flight control loop. The time it takes to do the face detection slows the speed of the main loop and therefore how responsive the code is to input from the controller. This is not really a desirable situation and we will explore a solution in the next lesson.