Challenge 1 for MBZIRC
ROS wrapper for the mbzirc_estimation_lib
catkin build mbzirc_target_tracking --force-cmake
Make sure to init refill dependency as submodule: git submodule update --init --recursive
To launch the estimator use or adopt mbzirc_target_tracking/launch/tracker_default.launch. Be sure to check or update the following settings which are set in the launchfile:
- Ensure that /tf for the frame_ids camera and target are broadcasted
- Adopt hand_eye_calibration.yaml with the transform between observer odometry frame and camera frame
- Add estimator settings: For each new estimator create a config.yaml in mbzirc_target_tracking/config and specify which type of estimator, which measurement model and which process models to use. Check mbzirc_target_tracking/config/imm_config.yaml for reference
- Add measurement model settings, check mbzirc_target_tracking/config/measurement_model_config.yaml for referecńce
In order to test the estimator in a simulation setup please use mbzirc_target_tracking/launch/tracker_gazebo.launch
roslaunch mbzirc_target_tracking tracker_gazebo.launch launch_gazebo:=true
Python node that loads tf.keras models trained with mbzirc_motion_prediction_lib to perform inference and predict future target positions A model checkpoint can be specified as rosparam in the launchfile of the inference node.
Library to estimate target state based on detections that are subjected to dropouts or noise. Input: 3D position measurements of the targets position. (If the measurements are not performed in world frame but from a moving observer you will need to broadcast the camera pose as /tf)
- Single Model
- Naive Multi-Model: Perform time update and measurement update with a set of filters/models. Determine the model which offered the most likely posterior and use it as overall state
- Naive Multi-Model: Perform time update and measurement update with a set of filters/models. Obtain estimates as probabilistic weighted mean of the submodel posteriors.
- Generalized Pseudo Bayesian Multi-Model (GPB): Reinitialize all filter states to a common weighted mean after each measurement update. Obtain estimates as probabilistic weighted mean of the submodel posteriors.
- Interacting Multi-Model (IMM) - Recommended: Re-initialize each model with a weighted average of all other model states given a transition probability graph. Obtain estimates as probabilistic weighted mean of the submodel posteriors. This guide gives a good overview on the background and algorithms behind different types of available multi-model estimators.
Supported motion models:
- Constant Position
- Constant Velocity
- Constant Acceleration
- Turn at constant radius
Additional to these motion models a RNN can be used as process model. Node mbzirc_motion_learning_inference allows to load a learned model, subscribe to states, perform inference and publish predicted states. mbzirc_motion_learning_lib holds the code to train and evaluate RNNs for target state prediction using tensorflow's keras API
The node mbzirc_target_tracking offers a ROS wrapper for mbzirc_estimation_lib and will handle all subscribing and publishing as well as loading all relevant params.
This library allows to train a Recurrent Neural Network (RNN) that takes in target positions to predict a finite number of future positions. Its hidden state allows the model to make use of input that is much further in the past to capture broader priors on the expected trajectory our long-term repetitiveness. The network consist of an encoder-decoder RNN. The encoder consists of a vanilla RNN of stacked GRU or LSTM units. For the decoder, two options are available:
- Recursive decoder - recommened: Initialize decoder state with the encoders hidden state (latent features). Refeed output of decoder unit N times back into itself to produce N state predictions. As first decoder input use zero state. More details here.
- Repeat decoder: Initialize decoder hidden input with zeros, and use the encoder's hidden state output as input to the decoder for every time-step. Thus, the encoder's hidden state is copied N times to be used as the N time series inputs for the decoder of length N. The output of the decoder unit at each step is then the predicted state. More details here.
If you have a saved model checkpoint that you want to use for inference you can use the node mbzirc_rnn_inference [Incoming PR] to load the model, subscribe states and publish predictions. You do not need to perform any prior setup except for building mbzirc_rnn_inference and launching it. Follow the next steps only if you want to re-train or diagnose the performance of the RNN.
- Navigate to the library root and create virtual environment that uses a python3 interpreter
virtualenv venv -p /usr/bin/python3.5 - Activate and install python dependencies
source venv/bin/activatepip install -r requirements.txt - Choose were you want to save dataset and experiment files and export the location as environment variable
export SCRATCH=<data_dir>
If you just want to check the performance of your model use the provided notebook jupyter
jupyter notebook notebooks/visualize_predictions.ipynb
If you run the notebook you will be prompted to choose a experiment from which to restore the model
All relevant paramters are passed as json file. Check the available .json in the config folder to get an overview of the available paramters. Among others you can specify the loss to be used, type of recurrent layers, data augmentation parameters, number of hidden units and more.
Make sure you activated the virtual environment. Navigate to this librarys root and invoke
python src/main.py -c config/<your_config.json>
If you want to resume training from a previous session:
python src/main.py -r <experiment_dir> -e <num_additional_epochs>
The model was trained using CSV files of recorded trajectories. The class DataloaderTrajectoryCSV offers an implementation for loading and feeding trajectory states from a set of CSV files. Depending on how the training dataset is structured you will have to write your own dataloader.py. Derive from dataloader_base.py and implement:
- generator_fn: A function that yields individual states one-by-one. It should yield a 0-D numpy array with the length of the state dimensionality
- build_dataset: Takes in generator_fn to create a tf.data.Datset. Add any desired transformation, augmentation or batching using the tf.data.map, tf.data.filter or tf.data.apply
contains the main launch files for quick access. Please note that launch files are also included in the individual nodes.
contains the parameter (.yaml) files called by the nodes
Use rosinstall to fetch dependencides:
cd ~/catkin_ws/src
wstool init
wstool merge mbzirc_2020_challenge_1/mbzirc_challenge1.rosinstall
wstool update
- refill (just use
git submodule update --init --recursiveto pull the feature/mbzirc_ch1 branch) - apriltag
- apriltag_ros
- mav_trajecory_generation
- yaml_cpp_catkin
- ceres solver