High School

Overview: 
Explore processing the Tello camera view to locate and then track ArUco marker if visible. This is an example of programming autonomous operation by the drone.
Objectives: 

Understand how to process camera view information to cause the drone to act autonmously based on what it "sees".

Content: 

This example program demonstrates how to program an autonomous behavior for the Tello, where it automatically acts on what it "sees". This program is a modification of the FindMarker program. In that program, we processed the video stream from the drone looking for the presence of an ArUco Marker in the camera field of view.

In this program when we enter marker detection mode with the X button, the program will take over flight control of the drone. It will rotate slowly until it detects an ArUco Marker in the camera field of view. The program will process the information returned by the marker detection class and compute a new flight command to center the marker image in the field of view. We do this by using the target rectangle. We determine the X axis location of the target center line and compare it to the image (field of view) center line. The difference tells us how far and in which direction the target rectangle is  offset  from the center of the field of view. We use this information to turn the drone as needed to point directly at the marker. As long as the marker is detected, the program will work to keep the marker centered and so cause the drone to track the marker. If marker detection is lost, the drone will resume rotating to acquire a new marker to track.

Additionally, when a  marker is first detected, the program records the size of the marker target rectangle. As it processes subsequent images it compares the new size of marker to the original size and uses the difference as a measure of distance. It then adds a forward or backward component to the flight command to make the drone maintain the original distance to the marker. If the target rectangle area is smaller than the original area, the target has moved away from the drone so we fly forward to get the target areas to match again. If the target area is larger than the original size, the target has moved closer to the drone so we fly backward to restore the original distance.

A key part of such schemes is to adjust (or scale) the offset and distance values to values that make sense as flyRC commands so that the drone moves at a speed that is compatible with the field of view and size of the marker. When trying to center the target, moving the drone too fast will cause the target to go out of view and stop the tracking. In the same vein, if you move the target too fast the drone will not be able to track it. All of this can be a challenge due to the Tello's small field of view.

Note: in order to improve tracking accuracy and responsiveness, we have shortened the delay of the main loop to 25ms.

Modify the Main class to execute the TrackMarker class and test autonomous marker tracking.

A great exercise for you would be to modify the program to adjust the drone's height to center the target vertically as well horizontally.

 

Navigation:

Track ArUco Marker

This example program demonstrates how to program an autonomous behavior for the Tello, where it automatically acts on what it "sees". This program is a modification of the FindMarker program. In that program, we processed the video stream from the drone looking for the presence of an ArUco Marker in the camera field of view.

Overview: 
Explore how to do the face detection function in a separate thread of execution thereby improving the responsiveness of the main program loop.
Objectives: 

Understand how using multi-threading can improve program responsiveness to user input.

Content: 

In the previous lesson we observed that running the face detection function in the program main loop slowed the programs response  to control inputs. Why is that? This happens because the face detection process is a relatively long process. The time taken to process images for faces lengthens the time to complete one main loop in the program. So the program sees and responds to control inputs less frequently. This is not a good situation in most any application but particularly when the program is controlling a machine. Good programming practice calls for keeping a programs main input processing loop as short and quick running as possible. So how can we do that?

The answer is using more than one thread in the program. A thread is a path of execution in your program (more here). The example programs, like any program, have a main thread started by the Java virtual machine to run your main() method. This thread starts when you run your program and everything in your program happens as a sequential action along that thread of execution. So every statement happens only when the previous statement is completed. That explains why the main loop slows down, because we added a lot of code (our own and OpenCV) to the main loop when we added face detection. You can see the difference in responsiveness when you turn face detection on and off with the X button. In Java, as in most programming languages, you can create additional threads of execution that can run in parallel to the main thread. You can use these other threads to execute code separately from the main thread thereby shortening its execution path and restoring it's responsiveness. So our solution to the control delay problem is to create a separate thread to run the face detection process. Now it is true that the new thread and the main thread compete for cpu time but there is typically enough cpu available to run several threads with cpu not being a constraining factor and changing the length of the main thread gains us far more than any cpu sharing issue. This is true with our example but be aware that in other applications competition for the cpu between threads may be an issue

So how do we go about moving the face detection to a new thread? Take a look at the FindFace2 example program. There are several ways to employ threads in a Java program. We will create a new class that extends the Java built-in Thread class. In Java a thread is represented as an object and so has a class we can extend to add the code we want to run in the thread. We will also use a technique called an inner class to create this thread class inside our existing program class. We could have created it as another external class in a separate file but since this new thread would only be used by our program it is simpler to just include it inside our drone program main class. You can see this thread class starting about line 220.

In its minimal form, our thread class (CheckForFaces) extends Thread and defines one method, run(). In the run method, we put our code to be executed by the thread. As you can see, it looks just like a cut down version of the main loop, a loop with just the face detection code in it.

To launch this thread object, we create a new instance of the CheckForFaces class and call it's run() method (line 140). Once we call run(), the thread will run our code until it ends of its own accord. Our code is a loop, so we need a way to stop it when we want to turn off face detection or end our program. We use the Thread class interrupt() method to signal the thread we want it to stop. Looking at the code in the thread run() method, we see its loop is conditioned on the Thread class built-in method isInterrupted(). This is how we watch for the signal to stop. Calling interrupt() causes isInterrupted() to return True.

Change the Main class to execute the FindFace2 class and observe the change in responsiveness gained by moving face detection to a separate thread.

A final note: in our example and pretty much any multi-threaded program, the threads we create will share some amount of global variables, static variables and object instances with the main thread and any other threads we might create. In simple cases like ours the sharing of these data resources is not an issue and we can ignore it. In more complex programs, the sharing of resources among threads becomes an important design consideration. The coordination of resource sharing is called concurrency and is a major area of Java and programming in general and beyond the scope of this curriculum. However, you can get a basic look at this by reading the TelloCamera class and looking for the Java synchronized statement. The camera class is the only significant shared resource in this particular program and that class was written to coordinate other threads and the video stream processing thread's access to images and the video stream.

 

Navigation:

Multi-Threaded Face Detection

In the previous lesson we observed that running the face detection function in the program main loop slowed the programs response  to control inputs. Why is that? This happens because the face detection process is a relatively long process. The time taken to process images for faces lengthens the time to complete one main loop in the program. So the program sees and responds to control inputs less frequently. This is not a good situation in most any application but particularly when the program is controlling a machine.

Overview: 
Explore using the OpenCV computer vision library to use the Tello camera to locate ArUco markers.
Objectives: 

Understand how the Tello-SDK uses of OpenCV computer vision library to process camera images sent by the Tello and further detect ArUco markers appearing in camera images. Understand how the programmer can make use of this feature.

Content: 

OpenCV is a free library of image and video processing functions freely available for anyone to use. It is fair to call it an industry standard. OpenCV contains many capabilities accessible at various levels of complexity, but by and large, it is complex and difficult to learn and use. To make it easier to use and more powerful, there are classes in OpenCV that add higher level functions over the base OpenCV API. All of this can make learning OpenCV a daunting task. However there are many tutorials and examples on the internet.

The Tello-SDK also provides a simple way to get started with OpenCV. It supports ArUco Marker detection with it's ArucoMarkers class. ArUco markers are small 2D barcodes. Each ArUco marker corresponds to a number, encoded into a grid of black and white pixels. The ArUco decoding algorithm is capable of locating, decoding, and estimating the pose (location and orientation in space) of any ArUco markers in the camera's field of view. ArUco markers are very useful as tags for many robotics and augmented reality applications. For example, one may place an ArUco next to a robot's charging station, an elevator button, or an object that a robot should manipulate. ArUco markers can be created with several standard dictionaries. Different dictionaries give rise to different numbers of pixels in the markers, and to different numbers of possible symbols that can be created using the dictionary. The default dictionary used by Tello-SDK is 4x4 with 50 symbols, numbered 1-50. In the Tello-SDK project folder, in the src/resources folder, is the file 15.png. This is an image of the number 15 ArUco marker. You can print it out and use it for testing.

ArUco marker detection uses OpenCV visual object detection functions to find ArUco markers in images. Using ArucoMarkers is a pretty easy way to get started using OpenCV. OpenCV has a set of functions specifically for working with ArUco markers.

This exploration of marker detection is intended to show what can be done with the Tello camera, which is really the Tello's only user programmable sensor. We can use OpenCV to process images from the Tello video stream and program the drone to respond to what it sees autonomously.

Take a look at the FindMarker example program. It is a modification of the FlyController program. You can see in the program main loop there is a block of code that is turned on or off with the X button. This block of code makes a call to the ArucoMarkers.detectMarkers() method. This method in turn makes the OpenCV calls needed to do a marker detection process on the current image from the drone video feed. The method returns true if one or more markers are detected.

If markers are detected, the code requests a count of the markers detected and then a list of Rectangle objects that define the location and size of the detected markers in the image. Finally it uses a feature of the TelloCamera class that allows the programmer to provide a list of Rectangles that will be drawn on the current image of the video feed until reset. This will draw boxes around the markers in the live video feed and any picture or recorded video.

You can also see that the program also requests the numeric id of the first detected marker. This is how code can determine what marker is being viewed by the drone.

Modify the Main class to execute the FindMarker class. Without taking off you can test the program with the drone in your hand using the printed out marker sample. You can then fly the drone and (carefully) fly it to point at the marker and test the marker detection.

The comments in the code will tell you more about what is going on and as always, you are encouraged to look at the MarkerDetection class code to see what is happening at the next lower level.

To learn more about OpenCV and image processing, view this basic tutorial. Here is more detailed OpenCV documentation and here is the JavaDoc.

 

Navigation:

ArUco Maker Detection with OpenCV

OpenCV is a free library of image and video processing functions freely available for anyone to use. It is fair to call it an industry standard. OpenCV contains many capabilities accessible at various levels of complexity, but by and large, it is complex and difficult to learn and use. To make it easier to use and more powerful, there are classes in OpenCV that add higher level functions over the base OpenCV API. All of this can make learning OpenCV a daunting task. However there are many tutorials and examples on the internet.

LEGO Gears Break Steel Rod

YouTube video shows how LEGO compound gearing-down set up breaks a steel rod.

https://m.youtube.com/watch?v=jRn5waE0qfk

Overview: 
Explore how to use the face detection feature of the OpenCV image processing library.
Objectives: 

Understand how face detection is implemented in the Tello-SDK and how to program the drone to detect and highlight faces it sees with it's camera.

Content: 

OpenCV is a free library of image and video processing functions freely available for anyone to use. It is fair to call it an industry standard. OpenCV contains many capabilities accessible at various levels of complexity, but by and large, it is complex and difficult to learn and use. To make it easier to use and more powerful, there are classes in OpenCV that add higher level functions over the base OpenCV API. All of this can make learning OpenCV daunting task. However there are many tutorials and examples on the internet.

The Tello-SDK also provides a simple way to get started with OpenCV. It supports human face detection with it's FaceDetection class. FaceDetection uses OpenCV image object detection functions to find faces in images. Using FaceDetection is a pretty easy way to get started using OpenCV.

This exploration of face detection is intended to show what can be done with the Tello camera, which is really the Tello's only user programmable sensor. We can use OpenCV to process images from the Tello video stream and program the drone to respond to what it sees autonomously.

Take a look at the FindFace example program. It is a modification of the FlyController program. You can see in the program main loop there is a block of code that is turned on or off with the X button. This block of code makes a call to the FaceDetection.detectFaces() method. This method in turn makes the OpenCV calls needed to do a face detection process on the current image from the drone video feed. The method returns true if one or more faces are detected.

If faces are detected, the code requests a count of the faces detected and then a list of Rectangle objects that define the location and size of the detected faces in the image. Finally it uses a feature of the TelloCamera class that allows the programmer to provide a list of Rectangles that will be drawn on the current image of the video feed until reset. This will draw boxes around the faces in the live video feed and any picture or recorded video.

Modify the Main class to execute the FindFace class. Without taking off you can test the program with the drone in your hand. You can then fly the drone and (carefully) fly it to point at a person and test the face detection.

The comments in the code will tell you more about what is going on and as always, you are encouraged to look at the FaceDetection class code to see what is happening at the next lower level.

To learn more about OpenCV and image processing, view this basic tutorial. Here is more detailed OpenCV documentation and here is the JavaDoc.

One thing you will notice is that the drone flight controls and buttons are not as responsive as before. This is due to the fact that the face detection processing is done in-line in the flight control loop. The time it takes to do the face detection slows the speed of the main loop and therefore how responsive the code is to input from the controller. This is not really a desirable situation and we will explore a solution in the next lesson.

 

Navigation:

Face Detection with OpenCV

OpenCV is a free library of image and video processing functions freely available for anyone to use. It is fair to call it an industry standard. OpenCV contains many capabilities accessible at various levels of complexity, but by and large, it is complex and difficult to learn and use. To make it easier to use and more powerful, there are classes in OpenCV that add higher level functions over the base OpenCV API. All of this can make learning OpenCV daunting task. However there are many tutorials and examples on the internet.

Overview: 
Explore how to fly the Tello under direct user control via a game controller.
Objectives: 

Understand the Tello's RC control command and how to write a program that allows direct user control of the drone.

Content: 

So far we have written drone control programs that are autonomous in nature. That is, they do what they do without any input from us in the course of execution. Lets switch gears and explore writing programs that allow for direct user control of the drone. The Tello has a command called RC (remote control) that accepts input from a game controller. Using this command and a controller, you can operate the Tello remotely just like a Tetrix or RoboRio based robot.

The Tello-SDK includes support for X-Box style controllers. This lesson was developed using a Logitech F310 controller just like Tetrix robots. When you plug your controller into a Windows PC the first time it will install a Windows driver for the controller.

With all this in place, we can write a program that controls drone movement with controller joysticks and drone functions with controller  buttons. At the start of the example, the code takes no flight action, but goes into a loop monitoring the controller buttons. The user controls the drone. You have to press the Start button to take off and press the Back button to land. You can press the A button to take a picture and the Dpad.Up button to do a flip. Use the joysticks to control drone movement. The amount of joy stick deflection controls the speed at which the drone moves.

Examine the FlyController class to see how all this is done. Then modify the Main class to execute the FlyController class.

Modify the code to support starting and stopping video recording. Add more Dpad buttons to select the other flip directions.

 

Navigation:

Pages