XSkill: Cross Embodiment Skill Discovery

^1,2Mengda Xu, ^1,Zhenjia Xu, ¹Cheng Chi, ^2,3Manuela Veloso, ¹Shuran Song

¹Columbia University, ²JP Morgan AI Research,³CMU

CoRL 2023

Project Page|arxiv

This repository contains code for training and evaluating XSkill in both simulation and real-world settings.

🚀 Installation

Follow these steps to install XSkill:

1. Create and activate the conda environment:

   cd xskill
   conda env create -f environment.yml
   conda activate xskill
   pip install -e . 

📦 Simulation Dataset

To set up the simulation dataset:

1. Create a new directory for datasets under XSkill:

   mkdir datasets
   cd datasets
   wget https://xskill.cs.columbia.edu/data/kitchen.zip
   unzip kitchen.zip

2. Set the base_dev_dir config/simulation/create_kitchen_datase.yaml to your working directory. Run the following command to generate the cross-embodiment kitchen data and the training mask:

   cd scripts
   python create_all_kitchen_dataset.py
   python extract_kitchen_info.py

🌐 Real World Dataset

To Download the real world kitchen dataset:

mkdir datasets
cd datasets
wget https://xskill.cs.columbia.edu/data/real_kitchen_data.zip

🚴‍♂️ Training

Simulation

1. Run the skill discovery script:

   python scripts/skill_discovery.py

2. Label the dataset using the learned prototype by the trained model.

   python scripts/label_sim_kitchen_dataset.py

3. Execute the skill transfer and composing script. Replace the pretrain_path and pretrain_ckpt in cfg/simulation/skill_transfer_composing.yaml

   python scripts/skill_transfer_composing.py
   

Real World

1. Execute the real-world skill discovery script:

   python scripts/realworld/skill_discovery.py

2. Label the real-world dataset:

   python scripts/realworld/label_real_kitchen_dataset.py

3. 📊 Visualization

   Open the provided Jupyter notebook viz_real.ipynb to visualize the learned prototypes:

BibTeX

@inproceedings{
     xu2023xskill,
     title={{XS}kill: Cross Embodiment Skill Discovery},
     author={Mengda Xu and Zhenjia Xu and Cheng Chi and  Manuela Veloso and Shuran Song},
     booktitle={7th Annual Conference on Robot Learning},
     year={2023},
     url={https://openreview.net/forum?id=8L6pHd9aS6w}
     }

License

This repository is released under the MIT license.

Acknowledgement

• Diffusion Policy is adapted from Diffusion Policy
• Many useful utilies are adapted from XIRL.