Sequential Learning for Dance generation

Generating dance using deep learning techniques.

The proposed model is shown in the following image:

The joints of the skeleton employed in the experiment are shown in the following image:

Use of GPU

If you use GPU in your experiment, set --gpu option in run.sh appropriately, e.g.,

$ ./run.sh --gpu 0

Default setup uses GPU 0 (--gpu 0). For CPU execution set gpu to -1

Execution

The main routine is executed by:

$ ./run.sh --net $net --exp $exp --sequence $sequence --epoch $epochs --stage $stage

Being possible to train different type of datasets ($exp)

To run into a docker container use the file (run_in_docker.sh) instead of (run.sh)

Unreal Engine 4 Visualization

For demostration from evaluation files or for testing training files use (local/ue4_send_osc.py). For realtime emulation execute (run_realtime.sh).

Requirements

For training and evaluating the following python libraries are required:

• chainer=>3.1.0
• chainerui
• cupy=>2.1.0
• madmom
• Beat Tracking Evaluation toolbox. The original code is found here
• mir_eval
• transforms3d
• h5py, numpy, soundfile, scipy, scikit-learn, pandas

Install the following music libraries to convert the audio files:

$ sudo apt-get install libsox-fmt-mp3

Additionally, you may require Marsyas to extract the bet reference information.

For real-time emulation:

• pyOSC (for python v2)
• python-osc (for python v3)
• vlc (optional)

ToDo:

• New dataset
• Detailed audio information
• Virtual environment release

Acknowledgement

• Thanks Johnson Lai for the comments

References

[1] Nelson Yalta, Shinji Watanabe, Kazuhiro Nakadai, Tetsuya Ogata, "Weakly Supervised Deep Recurrent Neural Networks for Basic Dance Step Generation", arXiv

[2] Nelson Yalta, Kazuhiro Nakadai, Tetsuya Ogata, "Sequential Deep Learning for Dancing Motion Generation", SIG-Challenge 2016