Skip to content

Latest commit

 

History

History
135 lines (95 loc) · 7.53 KB

README.md

File metadata and controls

135 lines (95 loc) · 7.53 KB
Raw
Matterport Mesh Unreal Engine / Carla Mesh

ViPlanner Omniverse Extension

IsaacSim Python Linux platform pre-commit License

The ViPlanner Omniverse Extension offers a testing environment for ViPlanner and includes the data collection pipeline. Within NVIDIA Isaac Sim as a photorealistic simulator and using IsaacLab, this extension provides an assessment tool for ViPlanner's performance across diverse environments.

Installation

To install the ViPlanner extension for Isaac Sim version 4.2.0, follow these steps:

  1. Clone the IsaacLab repo.

    git clone [email protected]:isaac-sim/IsaacLab.git

    and follow the installation steps given here to install both IsaacLab and IsaacSim.

    IMPORTANT: The extension is tested with IsaacSim 4.2.0 and IsaacLab 1.2.0 (commit: 0bccd886c19a9891c0b6bdd37e13f338eacc0bba). While it may work with newer versions, it is not guaranteed. If you encounter issues, please switch back to the previous versions. Any PRs with updated versions are welcome.

  2. Link the ViPlanner and Matterport extension to the IsaacLab repo.

    cd IsaacLab/source/extensions
ln -s <path-to-viplanner-repo>/omniverse/extension/omni.viplanner .
ln -s <path-to-viplanner-repo>/omniverse/extension/omni.isaac.matterport .

  3. Install ViPlanner and the extension into the virtual environment of IsaacLab.

    cd IsaacLab
./isaaclab.sh -p -m pip install -e <path-to-viplanner-repo>  # this install the ViPlanner package
./isaaclab.sh -i  # this installs the additional extensions

Remark It is necessary to comply with PEP660 for the install. This requires the following versions (as described here in detail)

  • pip >= 21.3 
    cd IsaacLab
./isaaclab.sh -p -m pip install --upgrade pip
  • setuptools >= 64.0.0 
    cd IsaacLab
./isaaclab.sh -p -m pip install --upgrade setuptools

Download the Simulation Environments

Matterport

To download Matterport datasets, please refer to the Matterport3D website. The dataset should be converted to USD format using Isaac Sim by executing the following steps:

  1. Run the convert_mesh.py script to convert the .obj file (located under matterport_mesh) to USD. With the recent update of the asset converter script, use the resulting *_non_metric.usd file.

    # run the converter
cd IsaacLab
./isaaclab.sh -p source/standalone/tools/convert_mesh.py matterport_mesh/xxx.obj matterport_mesh/xxx.usd --make-instanceable --collision-approximation convexDecomposition

    IMPORTANT

    • The conversion will fail if the asset name starts with a number (e.g., 0c334eaabb844eaaad049cbbb2e0a4f2.obj), because the USD API prohibits this naming as explained here. This issue is described in more detail here.

    • Typically, the file should be converted with correct scale and rotation. In the case that you run the demo and the environment is rotated or scaled incorrectly, fix the import setting such as Rotation and Scale. (Property -> Transform -> Rotate:unitsResolve = 0.0; Scale:unitsResolve = [1.0, 1.0, 1.0]) and then save the USD.

  2. Check that the converted mesh has colliders enabled. This can be done by starting IsaacSim (./isaaclab.sh -s), importing the converted usd mesh and visualizing the colliders (go to the "eye" symbol in the top left corner, select Show by Type -> Physics -> Colliders and set the value to All ). The colliders should be visible as pink linkes. In the case that no colliders are presented, select the mesh in the stage, go the Property section and click Add -> Physics -> Colliders Preset. Then save the asset.

Carla

We provide an already converted asset of the Town01 of Carla. It can be downloaded as USD asset: Download USD Link

Planer Demo

A demo script is provided to run the planner in three different environments: Matterport, Carla, and NVIDIA Warehouse. In each scenario, the goal is represented as a movable cube within the environment.

To run the demo, download the model: [checkpoint] [config] and the environment files. Then adjust the paths (marked as ${USER_PATH_TO_USD}) in the corresponding config files.

Matterport

Config

The demo uses the 2n8kARJN3HM scene from the Matterport dataset. A preview is available here.

cd IsaacLab
./isaaclab.sh -p <path-to-viplanner-repo>/omniverse/standalone/viplanner_demo.py --scene matterport --model_dir <path-to-model-download-dir>

Carla

Config

cd IsaacLab
./isaaclab.sh -p <path-to-viplanner-repo>/omniverse/standalone/viplanner_demo.py --scene carla --model_dir <path-to-model-download-dir>

NVIDIA Warehouse

No need to adjust the config, as the asset is directly included in the repo.

cd IsaacLab
./isaaclab.sh -p <path-to-viplanner-repo>/omniverse/standalone/viplanner_demo.py --scene warehouse --model_dir <path-to-model-download-dir>

Data Collection

The training data is generated from different simulation environments. After they have been downloaded and converted to USD, adjust the paths (marked as ${USER_PATH_TO_USD}) in the corresponding config files (Carla and Matterport). The rendered viewpoints are collected by executing

cd IsaacLab
./isaaclab.sh -p <path-to-viplanner-repo>/omniverse/standalone/data_collect.py --scene <matterport/carla/warehouse> --num_samples <how-many-viewpoints>

To test that the data has been correctly extracted, please run the 3D reconstruction and see if the results fits to the simulated environment.