In this project, a deep neural network model is trained to identify and track a target in simulation. Follow Me applications like this are key to many fields of robotics and the very same techniques applied here, could be extended to scenarios like advanced cruise control in autonomous vehicles or human-robot collaboration in the industry.
Install Dependencies
Following frameworks and package dependencies are must to the project for up and running into the system:
- Python 3.x
- Tensorflow 1.2.1
- NumPy 1.11
- SciPy 0.17.0
- eventlet
- Flask
- h5py
- PIL
- python-socketio
- scikit-image
- transforms3d
- PyQt4/Pyqt5
With the training and validation data having been generated or downloaded, the Project is ready with Neural Net training.
Note: Training CNNs is a very compute-intensive process. If the system does not have a recent Nvidia graphics card, with cuDNN and CUDA installed , it may be needed to perform the training step in the cloud. Instructions for using AWS to train the network in the cloud may be found here
To score the network on the Follow Me task, two types of error are measured. First the intersection over the union for the pixelwise classifications is computed for the target channel.
In addition to this we determine whether the network detected the target person or not. If more then 3 pixels have probability greater then 0.5 of being the target person then this counts as the network guessing the target is in the image.
We determine whether the target is actually in the image by whether there are more then 3 pixels containing the target in the label mask.
Using the above the number of detection true_positives, false positives, false negatives are counted.
How the Final score is Calculated
The final score is the pixelwise average_IoU*(n_true_positive/(n_true_positive+n_false_positive+n_false_negative)) on data similar to that provided in sample_evaulation_data
- Copy the saved model to the weights directory
data/weights. - Launch the simulator, select "Spawn People", and then click the "Follow Me" button.
- Run the realtime follower script
$ python follower.py my_amazing_model.h5
The schematic representation of the completed training model looks like:
The performance of the trained model for “Images while following the target” over 3 random images looks like:
The performance of the trained model for “Images while at patrol without target” over 3 random images looks like:
