Skip to content

Commit

Permalink
Support for Isaac Orbit; ./repair utility; updated drivers and IS; bu…
Browse files Browse the repository at this point in the history
…gfixes; more mapped folders; doc improvements; more new and improved bugs.
  • Loading branch information
myurasov committed Dec 21, 2023
2 parents 187d372 + 5f90b28 commit edf0aa9
Show file tree
Hide file tree
Showing 26 changed files with 490 additions and 31 deletions.
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -92,4 +92,4 @@ WORKDIR /app

ENTRYPOINT [ "/bin/sh", "-c" ]

ENV VERSION="v2.1.1"
ENV VERSION="v2.2.0"
58 changes: 52 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,13 @@
- [Running Applications](#running-applications)
- [Isaac Sim](#isaac-sim)
- [Shell in Isaac Sim Container](#shell-in-isaac-sim-container)
- [Omniverse Isaac Gym](#omniverse-isaac-gym)
- [Omniverse Isaac Gym Environments](#omniverse-isaac-gym-environments)
- [Isaac Orbit](#isaac-orbit)
- [Mapped Folders](#mapped-folders)
- [Pausing and Resuming](#pausing-and-resuming)
- [Uploading Data](#uploading-data)
- [Downloading Data](#downloading-data)
- [Repairing](#repairing)
- [Destroying](#destroying)

This tool automates deployment of [Isaac Sim](https://developer.nvidia.com/isaac-sim) to public clouds.
Expand Down Expand Up @@ -199,18 +202,50 @@ To get a shell inside Isaac Sim container, click "Isaac Sim Shell" icon on the d
~/Desktop/isaacsim-shell.sh
```

#### Omniverse Isaac Gym
#### Omniverse Isaac Gym Environments

[Omniverse Isaac Gym Reinforcement Learning Environments for Isaac Sim](https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs) ("Omniverse Isaac Gym") is pre-installed on the deployed Isaac instances.
[Omniverse Isaac Gym Reinforcement Learning Environments for Isaac Sim](https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs) ("Omni Isaac Gym Envs") can be pre-installed on the deployed Isaac instances.

To run Omniverse Isaac Gym click "Omni Isaac Gym" icon on the desktop or run the following command in the terminal:
To run Omniverse Isaac Gym Environments click "Omni Isaac Gym Envs" icon on the desktop or run the following command in the terminal:

```sh
~/Desktop/omni-isaac-gym-envs.sh
```

Default output directory (`/OmniIsaacGymEnvs/omniisaacgymenvs/runs`) in the OmniIsaacGymEnvs contaner will be linked to the default results directory (`/home/ubuntu/results`) on the deployed instance. You can download the contents of this directory to your local machine using `./download <deployment_name>` command.

Tip: To install a specific git reference of OmniIsaacGymEnvs, provide valid reference from <https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs> as a value of `--oige` parameter to the deployment command. For example, to install `devel` branch on an AWS instance, run the following command:

```sh
./deploy-aws --oige devel
```

#### Isaac Orbit

*Isaac Orbit is still experimental and intended for preview purposes only.*

[Isaac Orbit](https://isaac-orbit.github.io/orbit/index.html) can be pre-installed on the deployed Isaac instances.

To run Isaac Orbit click "Isaac Orbit" icon on the desktop or run the following command in the terminal:

```sh
~/Desktop/isaac-orbit.sh
```

Tip: To install a specific git reference of Isaac Orbit, provide valid git reference from <https://github.com/NVIDIA-Omniverse/Orbit> as a value of `--orbit` parameter to the deployment command. For example, to install `devel` branch on an AWS instance, run the following command:

```sh
./deploy-aws --orbit devel
```

### Mapped Folders

The following folders are mapped to the running Isaac Sim container by default (container paths may be different for specific applications):

- `/home/ubuntu/uploads` (host) --> `/uploads` (container) - user data uploaded to the deployment with `./upload` command or automatically from local `uploads/` folder
- `/home/ubuntu/results` (host) --> `/results` (container) - results of the applications run on the deployment, can be downloaded from the deployed machine with `./download` command
- `/home/ubuntu/workspace` (host) --> `/workspace` (container) - workspace folder, can be used to exchange data between the host and the container.

### Pausing and Resuming

You can stop and re-start instances to save on cloud costs. To do so, run the following commands:
Expand All @@ -223,8 +258,6 @@ You can stop and re-start instances to save on cloud costs. To do so, run the fo
./start <deployment-name>
```

Currently, stop-start is only supported for Azure deployments, other clouds will be added soon.

### Uploading Data

You can upload user data from `uploads/` folder (in the project root) to the deployment by running the following command:
Expand All @@ -251,6 +284,19 @@ You can download user data to `results/` folder (in the project root) from deplo

Data will be downloaded from `/home/ubuntu/results` directory by default. You can change this by passing `--remote-dir` argument to the command. Run `./download --help` to see more options.

### Repairing

If for some reason the deployment cloud resouces or software configuration get corrupted, you can attempt to repair the deployment by running the following command:

```sh
# run both terraform and ansible
./repair <deployment-name>
# just run terraform to try fixing the cloud resources
./repair <deployment-name> --no-ansible
# just run ansible to try fixing the software configuration
./repair <deployment-name> --no-terraform
```

### Destroying

To destroy a deployment, run the following command:
Expand Down
112 changes: 112 additions & 0 deletions repair
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env python3

# region copyright
# Copyright 2023 NVIDIA Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# endregion


import os
import sys

import click

from src.python.config import c as config
from src.python.utils import (
colorize_error,
colorize_info,
deployments,
format_cloud_name,
read_meta,
read_tf_output,
shell_command,
)


@click.command()
@click.option(
"--debug/--no-debug",
default=False,
show_default=True,
)
@click.argument(
"deployment_name",
required=True,
type=click.Choice(deployments()),
)
@click.option(
"--terraform/--no-terraform",
"-t/-T",
default=True,
show_default=True,
help="Run Teraform?",
)
@click.option(
"--ansible/--no-ansible",
"-a/-A",
default=True,
show_default=True,
help="Run ansible?",
)
def main(
debug: bool,
deployment_name: str,
terraform: bool,
ansible: bool,
):
meta = read_meta(deployment_name, debug)
cloud = read_tf_output(deployment_name, "cloud", verbose=debug)

click.echo(
colorize_info(
f"* Repairing deployment '{deployment_name}' @ {format_cloud_name(cloud)}..."
)
)

# run terraform
if terraform:
click.echo(colorize_info("* Re-running Terraform..."))
shell_command(
f"terraform init -upgrade -no-color -input=false {' > /dev/null' if not debug else ''}",
cwd=f"{config['terraform_dir']}/{cloud}",
verbose=debug,
)
shell_command(
"terraform apply -auto-approve "
+ f"-state={config['state_dir']}/{deployment_name}/.tfstate "
+ f"-var-file={config['state_dir']}/{deployment_name}/.tfvars",
cwd=f"{config['terraform_dir']}/{cloud}",
verbose=debug,
)

# run ansible
if ansible:
click.echo(colorize_info("* Re-running Ansible..."))
for app in ["isaac", "ovami"]:
if meta["params"][app]:
shell_command(
f"ansible-playbook -i {config['state_dir']}/{deployment_name}/.inventory "
+ f"{app}.yml {'-vv' if debug else ''}",
cwd=f"{config['ansible_dir']}",
verbose=debug,
)


if __name__ == "__main__":
if os.path.exists("/.dockerenv"):
# we're in docker, run command
main()
else:
# we're outside, start docker container first
shell_command(f"./run '{' '.join(sys.argv)}'", verbose=True)
7 changes: 7 additions & 0 deletions src/ansible/inventory.template
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,17 @@ ansible_user="ubuntu"
ngc_api_key="{ngc_api_key}"
isaac_image="{isaac_image}"
vnc_password="{vnc_password}"
orbit_git_checkpoint="{orbit}"
omniverse_user="{omniverse_user}"
deployment_name="{deployment_name}"
omniverse_password="{omniverse_password}"
gcp_driver_url="{config[gcp_driver_url]}"
omni_isaac_gym_envs_git_checkpoint="{oige}"
system_user_password="{system_user_password}"
uploads_dir="{config[default_remote_uploads_dir]}"
results_dir="{config[default_remote_results_dir]}"
workspace_dir="{config[default_remote_workspace_dir]}"
generic_driver_apt_package="{config[generic_driver_apt_package]}"
ansible_ssh_private_key_file="{config[state_dir]}/{deployment_name}/key.pem"

[targets:children]
Expand Down
1 change: 0 additions & 1 deletion src/ansible/isaac.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@
in_china: False
application_name: isaac
prompt_ansi_color: 36 # cyan
nvidia_generic_driver_package: nvidia-driver-525-server
roles:
- isaac
handlers:
Expand Down
1 change: 0 additions & 1 deletion src/ansible/ovami.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
in_china: False
application_name: ovami
prompt_ansi_color: 34 # bright blue
nvidia_driver_package: nvidia-driver-525-server
roles:
- ovami
handlers:
Expand Down
9 changes: 8 additions & 1 deletion src/ansible/roles/isaac/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,21 @@ isaac_image:

isaac_cache_dir: /home/{{ ansible_user }}/isaac

# directory for installing OmniIsaacGymEnvs assets (dockerfile, etc.)
# OmniIsaacGymEnvs
omni_isaac_gym_envs_dir: /opt/omni-isaac-gym-envs
omni_isaac_gym_envs_git_checkpoint: main

# Orbit
orbit_dir: /opt/orbit
orbit_git_checkpoint: devel

# "none" skips login/pull
ngc_api_key:

# directory to output results to
results_dir: /home/{{ ansible_user }}/results
workspace_dir: /home/{{ ansible_user }}/results
uploads_dir: /home/{{ ansible_user }}/uploads

omniverse_user:
omniverse_password:
Expand Down
2 changes: 1 addition & 1 deletion src/ansible/roles/isaac/files/omni-isaac-gym-envs.desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
Type=Application
Terminal=true
Exec=/home/ubuntu/Desktop/omni-isaac-gym-envs.sh
Name=Omni Isaac Gym
Name=Omni Isaac Gym Envs
Icon=/home/ubuntu/Pictures/isaacsim-icon-shell.png
6 changes: 6 additions & 0 deletions src/ansible/roles/isaac/files/orbit.desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[Desktop Entry]
Type=Application
Terminal=true
Exec=/home/ubuntu/Desktop/orbit.sh
Name=Isaac Orbit
Icon=/home/ubuntu/Pictures/isaacsim-icon-shell.png
9 changes: 8 additions & 1 deletion src/ansible/roles/isaac/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,14 @@

# https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs
- name: Omni Isaac Gym Envs
import_tasks: omni-isaac-gym-envs.yml
import_tasks: omni_isaac_gym_envs.yml
when: omni_isaac_gym_envs_git_checkpoint != 'no'

# https://isaac-orbit.github.io/orbit/index.html
- name: Orbit
import_tasks: orbit.yml
when: orbit_git_checkpoint != 'no'
tags: __orbit

- name: Restart NX server
meta: noop
Expand Down
73 changes: 73 additions & 0 deletions src/ansible/roles/isaac/tasks/orbit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# region copyright
# Copyright 2023 NVIDIA Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# endregion

- name: Create directory for Orbit
file:
path: "{{ orbit_dir }}"
state: directory
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: 0755

- name: Upload Orbit setup files [1]
template: src={{ item }}
dest="{{ orbit_dir }}/{{ item }}"
mode=755
owner=ubuntu
group=ubuntu
with_items:
- orbit.dockerfile
tags: __orbit_dockerfile

- name: Upload Orbit setup files [2]
template: src="{{ item }}"
dest="/home/{{ ansible_user }}/Desktop"
mode=755
owner=ubuntu
group=ubuntu
with_items:
- orbit.sh

- name: Desktop icon for Orbit
copy:
src: "{{ item }}"
dest: "/home/{{ ansible_user }}/Desktop/{{ item }}"
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
mode: 0644
with_items:
- orbit.desktop

- name: Allow execution of desktop icon for Orbit
shell: gio set "/home/{{ ansible_user }}/Desktop/{{ item }}" metadata::trusted true
become_user: "{{ ansible_user }}"
with_items:
- orbit.desktop

- name: Set permissions for Orbit desktop icon
file:
path: "/home/{{ ansible_user }}/Desktop/{{ item }}"
mode: 0755
owner: "{{ ansible_user }}"
group: "{{ ansible_user }}"
with_items:
- orbit.desktop

- name: Build Orbit
shell: docker build -t orbit -f "{{ orbit_dir }}/orbit.dockerfile" "{{ orbit_dir }}"
become_user: "{{ ansible_user }}"
tags:
- skip_in_ovami
Loading

0 comments on commit edf0aa9

Please sign in to comment.