Instructional Material: ArUco Maker Detection with OpenCV
OpenCV is a library of image and video processing functions freely available for anyone to use. It is fair to call it an industry standard. OpenCV contains many capabilities accessible at various levels of complexity, but by and large, it is complex and difficult to learn and use. To make it easier to use and more powerful, there are classes in OpenCV that add higher level functions over the base OpenCV API. All of this can make learning OpenCV a daunting task. However there are many tutorials and examples on the internet.
The Tello-SDK also provides a simple way to get started with OpenCV. It supports ArUco Marker detection with it's ArucoMarkers class. ArUco markers are small 2D barcodes. Each ArUco marker corresponds to a number, encoded into a grid of black and white pixels. The ArUco decoding algorithm is capable of locating, decoding, and estimating the pose (location and orientation in space) of any ArUco markers appearing in an image. ArUco markers are very useful as tags for many robotics and augmented reality applications. For example, one may place an ArUco next to a robot's charging station, an elevator button, or an object that a robot should manipulate. ArUco markers can be created with several standard dictionaries. Different dictionaries give rise to different numbers of pixels in the markers, and to different numbers of possible symbols that can be created using the dictionary. The default dictionary used by Tello-SDK is 4x4 with 50 symbols, numbered 1-50. In the Tello-SDK project folder, in the src/resources folder, is the file 15.png. This is an image of the number 15 ArUco marker. You can print it out and use it for testing.
ArUco marker detection uses OpenCV visual object detection functions to find ArUco markers in an image. Using ArucoMarkers is a pretty easy way to get started using OpenCV as OpenCV has a set of functions specifically for working with ArUco markers.
This exploration of marker detection is intended to show what can be done with the Tello camera, which is really the Tello's only user programmable sensor. We can use OpenCV to process images from the Tello video stream and program the drone to respond to what it sees autonomously.
Take a look at the FindMarker example program. It is a modification of the FlyController program. You can see in the program main loop there is a block of code that is turned on or off with the X button. This block of code makes a call to the ArucoMarkers.detectMarkers() method. This method in turn makes the OpenCV calls needed to do a marker detection process on the current image from the drone video feed. The method returns true if one or more markers are detected.
If markers are detected, the code requests a count of the markers detected and then a list of Rectangle objects that define the location and size of the detected markers in the image. Finally it uses a feature of the TelloCamera class that allows the programmer to provide a list of Rectangles that will be drawn on the current image of the video feed until reset. This will draw boxes around the markers in the live video feed and any picture or recorded video.
You can also see that the program also requests the numeric id of the first detected marker. This is how code can determine what marker is being viewed by the drone.
Modify the Main class to execute the FindMarker class. Without taking off you can test the program with the drone in your hand using the printed out marker sample. You can then fly the drone and (carefully) fly it to point at the marker and test the marker detection.
The comments in the code will tell you more about what is going on and as always, you are encouraged to look at the MarkerDetection class code to see what is happening at the next lower level.
To learn more about OpenCV and image processing, view this basic tutorial. Here is more detailed OpenCV documentation and here is the JavaDoc.
