---

Pushing Assisted Placing on a Tabletop

In this project, we train a pushing policy for a Franka Emika Panda arm to facilitate 2D dense object placement. The network accepts a 2D Truncated Signed Distance Function (TSDF) and an occupancy grid of the desktop and outputs a pixel coordinate and a pushing direction for a pushing action to make space on the desktop. Models for this project are trained using Omniverse Isaac Sim 2022.2.1 and Isaac Orbit v0.1.0. ROS_package_pushing is the ROS package of executing the PPO pushing and placing method with a robotic arm.

Installation

Please refer the Isaac Orbit documentation to learn more about the installation of Isaac Sim and Isaac Orbit.We ran this code on Isaac Sim 2022.2.1. Therefore, to execute this code, please install Isaac Sim 2022.2.1 to ensure there are no compatibility issues.

When istalling Isaac Orbit, please clone the orbit repository here, instead of official Isaac Orbit repository. After setting up the environment following the link, please install open3d, opencv-python, shapely in the created virtual environment.

Trained Models

The trained models for this project can be downloaded from trained models.

Network Training and Performance Visualization for Pushing

To visualize the performance of the pushing baseline method, please run:

./orbit.sh -p source/standalone/workflows/baseline/play.py --num_envs 1 --task Isaac-Push-50-baseline-Franka-v0

To train the framework of the PPO with CNN method, please run:

./orbit.sh -p source/standalone/workflows/sb3_customized/train.py --num_envs 1 --task Isaac-Push-50-PPO-Franka-v0 --headless 
 --save_path ./logs/ # save_path indicates the path of the directory to save the trained weight

To visualize the performance of the trained model of the PPO with CNN method, please run:

./orbit.sh -p source/standalone/workflows/sb3_customized/play.py --num_envs 1 --task Isaac-Push-50-PPO-val-Franka-v0 --checkpoint ./path # checkpoint indicates the path to the trained model

To train the framework of the DQN with FCN method without mask, please run:

./orbit.sh -p source/standalone/workflows/FCN_method/FCN_without_mask/train.py --num_envs 1 --task Isaac-Push-50-FCN-without-mask-train-Franka-v0  --headless --save_path ./logs/ # save_path indicates the path of the directory to save the trained weight

To visualize the performance of the trained model of the DQN with FCN method without mask, please run:

./orbit.sh -p source/standalone/workflows/FCN_method/FCN_without_mask/play.py --num_envs 1 --task Isaac-Push-50-FCN-val-Franka-v0 --checkpoint ./path # checkpoint indicates the path to the trained model weight

To train the framework of the DQN with FCN method with mask, please run:

./orbit.sh -p source/standalone/workflows/FCN_method/FCN_with_mask/train.py --num_envs 1 --task Isaac-Push-50-FCN-with-mask-train-Franka-v0 --headless --save_path ./logs/ # save_path indicates the path of the directory to save the trained weight

To visualize the performance of the trained model of the DQN with FCN method with mask, please run:

./orbit.sh -p source/standalone/workflows/FCN_method/FCN_with_mask/play.py --num_envs 1 --task Isaac-Push-50-FCN-val-Franka-v0 --checkpoint ./path # checkpoint indicates the path to the trained model weight

Data Generation & Placing

To generate clutter table scenes or visualize the proposed placing method, please run:

./orbit.sh -p source/standalone/thesis/generate_cluttered_scene.py --headless ##to vusialize the placing performance please run the code without '--healess'

Robot pushing and placing

This repository contains the ROS package of executing the PPO pushing and placing method with a Franka Emika Panda arm and a wrist-mounted Intel Realsense D435.

To run the scripts in the ROS package, please configure the workspace following the instructions. Then please check the document to run the code.