Instructional Material: Multi-Threaded Face Detection

In the previous lesson we observed that running the face detection function in the program main loop slowed the response to control inputs. Why is that? This happens because the face detection process is a relatively long process. The time taken to process images for faces lengthens the time to complete one main loop in the program. So the program sees and responds to control inputs less frequently. This is not a good situation in most any application but particularly when the program is controlling a machine. Good programming practice calls for keeping a program's main input processing loop as short and quick running as possible. So how can we do that?

The answer is using more than one thread in the program. A thread is a path of execution in your program (more here). The example programs, like any program, have a main thread started by the Java virtual machine to run your main() method. This thread starts when you run your program and everything in your program happens as a sequential action along that thread of execution. So every statement happens only when the previous statement is completed. That explains why the main loop slows down, because we added a lot of code (our own and OpenCV) to the main loop when we added face detection. You can see the difference in responsiveness when you turn face detection on and off with the X button. In Java, as in most programming languages, you can create additional threads of execution that can run in parallel to the main thread. You can use these other threads to execute code separately from the main thread thereby shortening its execution path and restoring it's responsiveness. So our solution to the control delay problem is to create a separate thread to run the face detection process. Now it is true that the new thread and the main thread compete for cpu time but there is typically enough cpu available to run several threads with cpu not being a constraining factor and changing the length of the main thread gains us far more than any cpu sharing issue. This is true with our example but be aware that in other applications competition for the cpu between threads may be an issue

So how do we go about moving the face detection to a new thread? Take a look at the FindFace2 example program. There are several ways to employ threads in a Java program. We will create a new class that extends the Java built-in Thread class. In Java a thread is represented as an object and so has a class we can extend to add the code we want to run in the thread. We will also use a technique called an inner class to create this thread class inside our existing program class. We could have created it as another external class in a separate file but since this new thread would only be used by our program it is simpler to just include it inside our drone program main class. You can see this thread class starting about line 220.

In its minimal form, our thread class (CheckForFaces) extends Thread and defines one method, run(). In the run method, we put our code to be executed by the thread. As you can see, it looks just like a cut down version of the main loop, a loop with just the face detection code in it.

To launch this thread object, we create a new instance of the CheckForFaces class and call it's run() method (line 140). Once we call run(), the thread will run our code until it ends of its own accord. Our code is a loop, so we need a way to stop it when we want to turn off face detection or end our program. We use the Thread class interrupt() method to signal the thread we want it to stop. Looking at the code in the thread run() method, we see its loop is conditioned on the Thread class built-in method isInterrupted(). This is how we watch for the signal to stop. Calling interrupt() causes isInterrupted() to return True.

Change the Main class to execute the FindFace2 class and observe the change in responsiveness gained by moving face detection to a separate thread.

A final note: in our example and pretty much any multi-threaded program, the threads we create will share some amount of global variables, static variables and object instances with the main thread and any other threads we might create. In simple cases like ours the sharing of these data resources is not an issue and we can ignore it. In more complex programs, the sharing of resources among threads becomes an important design consideration. The coordination of resource sharing is called concurrency and is a major area of Java and programming in general and beyond the scope of this curriculum. However, you can get a basic look at this by reading the TelloCamera class and looking for the Java synchronized statement. The camera class is the only significant shared resource in our Tello program. That class was written to coordinate access to images and the video feed between the video stream processing thread and the main (or any other) thread. The main idea behind concurrency is preventing more than one thread from manipulating the same data item at the same time.