How to run the simulation on PyBullet without GUI (Running on cloud headless)

[vishal-2000]

Hi,
Is there a way to run the OCRTOC baseline or any other solution on PyBullet simulator without GUI? We tried some stuff, and were able to switch off PyBullet GUI and RVIZ. But, the simulator gets stuck while executing the 2nd or 3rd trajectory. This happened with both the solution.launch approach (Running simulator, solution.launch and trigger.launch) and the recorded trajectories approach (Playing trajectories via .bag file). We, in order to run it headless, did the following:

1. Changed GUI=True to GUI=False in pybullet_simulator/scripts/load_env_ros.py. The above change tries to connect PyBullet to the SHARED_MEMORY server. But it seemed like, it failed to connect to SHARED_MEMORY.
2. So we added the following snippet to pybullet_simulator/scripts/pybullet_env.py to automatically connect PyBullet to DIRECT server whenever SHARED_MEMORY fails

if GUI is True:
  self.physics_client = pybullet.connect(pybullet.GUI_SERVER)
else:
  self.physics_client = pybullet.connect(pybullet.SHARED_MEMORY_SERVER)
  if self.physics_client < 0:
      self.physics_client = pybullet.connect(pybullet.DIRECT)

3. Then we commented line 3 in panda_moveit_config/launch/ocrtoc.launch to switch off RVIZ
4. We encountered some TIMEOUT errors after step 3, so for that, we added the following lines in panda_moveit_config/launch/move_group.launch to prevent TIMEOUT based errors:

<param name="move_group/trajectory_execution/allowed_execution_duration_scaling" value="4.0"/>
 <param name="move_group/trajectory_execution/execution_duration_monitoring" value="false"/>

5. After step 4, it was able to execute 2 to 3 trajectories, after that it got stuck with some execution. And it stayed there like forever, had to kill it.
   The following is what we got on playing .bag file (which contains the recorded trajectories) with roslaunch ocrtoc_task bringup_simulator_pybullet.launch task_index:=0-0
   roslaunch ocrtoc_task bringup_simulator_pybullet.launch task_index:=0-0 Output

root@marley:/# roslaunch ocrtoc_task bringup_simulator_pybullet.launch task_index:=0-0  
... logging to /root/.ros/log/029a808e-30a4-11ec-91f9-b881984c65d0/roslaunch-marley-1957.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
xacro: in-order processing became default in ROS Melodic. You can drop the option.
started roslaunch server http://marley:41637/
SUMMARY
========
PARAMETERS
 * /joint_state_controller/publish_rate: 30
 * /joint_state_controller/type: joint_state_contr...
 * /position_joint_trajectory_controller/constraints/goal_time: 0.5
 * /position_joint_trajectory_controller/constraints/panda_joint1/goal: 0.05
 * /position_joint_trajectory_controller/constraints/panda_joint2/goal: 0.05
 * /position_joint_trajectory_controller/constraints/panda_joint3/goal: 0.05
 * /position_joint_trajectory_controller/constraints/panda_joint4/goal: 0.05
 * /position_joint_trajectory_controller/constraints/panda_joint5/goal: 0.05
 * /position_joint_trajectory_controller/constraints/panda_joint6/goal: 0.05
 * /position_joint_trajectory_controller/constraints/panda_joint7/goal: 0.05
 * /position_joint_trajectory_controller/joints: ['panda_joint1', ...
 * /position_joint_trajectory_controller/type: position_controll...
 * /pybullet_env/task_index: 0-0
 * /robot_description: <?xml version="1....
 * /robot_state_publisher/publish_frequency: 50.0
 * /robot_state_publisher/tf_prefix: 
 * /rosdistro: melodic
 * /rosversion: 1.14.11
NODES
  /
    joint_state_controller_spawner (controller_manager/controller_manager)
    position_joint_trajectory_controller_spawner (controller_manager/controller_manager)
    pybullet_env (pybullet_simulator/load_env_ros.py)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    ros_interface (pybullet_simulator/ros_interface.py)
ROS_MASTER_URI=http://localhost:11311
process[pybullet_env-1]: started with pid [1978]
pybullet build time: Sep 13 2020 23:56:31
startThreads creating 1 threads.
starting thread 0
started thread 0 
PosixSharedMemory::releaseSharedMemory removed shared memoryPosixSharedMemory::releaseSharedMemory detached shared memory
PosixSharedMemory::releaseSharedMemory removed shared memoryPosixSharedMemory::releaseSharedMemory detached shared memory
MotionThreadFunc thread started
Floor friction updated!
('lateralFriction:', 1.0)
('spinningFriction:', 1.0)
('package path', '/root/ocrtoc_ws/src/pybullet_simulator')
b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link0b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link1b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link2b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link3b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link4b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link5process[ros_interface-2]: started with pid [1979]
b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link6b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link7b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_link8b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_handb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_leftfingerb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_rightfingerb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
realsense_depth_optical_frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
tableb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
panda_ee_linkb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
opengl_rgb_camera_linkb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
rgb_camera_linkb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
depth_camera_linkpybullet build time: Sep 13 2020 23:56:31
process[robot_state_publisher-3]: started with pid [1991]
process[joint_state_controller_spawner-4]: started with pid [1997]
process[position_joint_trajectory_controller_spawner-5]: started with pid [2004]
****************************************************************************************************
PyBullet Robot Info 
('connected to pybullet physics id:', 0)
****************************************************************************************************
('robot ID:              ', 1L)
('robot name:            ', u'panda')
('robot total mass:      ', 17.072002999999995)
('base link name:        ', u'panda_link0')
('num of joints:         ', 22)
('num of actuated joints:', 9)
('joint names:           ', 22, [u'panda_joint1', u'panda_joint2', u'panda_joint3', u'panda_joint4', u'panda_joint5', u'panda_joint6', u'panda_joint7', u'panda_joint8', u'panda_hand_joint', u'panda_finger_joint1', u'panda_finger_joint2', u'realsense_camera_joint', u'realsense_optical_joint', u'realsense_rgb2depth_virtual_joint', u'realsense_fake_joint', u'panda_hand_to_tool', u'panda_ee_link_joint', u'world_link', u'table_joint', u'kinect_camera_joint', u'kinect_opengl_virtual_joint', u'kinect_rgb2depth_virtual_joint'])
('joint indexes:         ', 22, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21])
('actuated joint names:  ', 9, [u'panda_joint1', u'panda_joint2', u'panda_joint3', u'panda_joint4', u'panda_joint5', u'panda_joint6', u'panda_joint7', u'panda_finger_joint1', u'panda_finger_joint2'])
('actuated joint indexes:', 9, [0, 1, 2, 3, 4, 5, 6, 9, 10])
('link names:            ', 22, [u'panda_link1', u'panda_link2', u'panda_link3', u'panda_link4', u'panda_link5', u'panda_link6', u'panda_link7', u'panda_link8', u'panda_hand', u'panda_leftfinger', u'panda_rightfinger', u'realsense_link', u'realsense_color_optical_frame', u'realsense_depth_optical_frame', u'realsense_camera_frame', u'tool', u'panda_ee_link', u'world', u'table', u'opengl_rgb_camera_link', u'rgb_camera_link', u'depth_camera_link'])
('link indexes:          ', 22, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21])
('joint dampings:        ', [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
('joint frictions:       ', [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
('positionContwrol', 'enabled!')
panda_leftfinger friction updated!
panda_rightfinger friction updated!
robot id:1, type:<type 'long'>
('world_path: ', '/root/ocrtoc_ws/src/ocrtoc_materials/scenes/0-0.world')
PosixSharedMemory::releaseSharedMemory detached shared memory
('scene_object_ids:', (2,))
/root/ocrtoc_ws/src/ocrtoc_materials/scenes/0-0.yaml
[INFO] [1634625989.603539]: number of joints of 1:22
{'orion_pie_v1': [0, 0, 0.1, 0, 0, 0]}
****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
('robot ID:              ', 1)
('robot name:            ', u'panda')
('robot total mass:      ', 17.072002999999995)
('base link name:        ', u'panda_link0')
('num of joints:         ', 22)
('num of actuated joints:', 9)
('joint names:           ', 22, [u'panda_joint1', u'panda_joint2', u'panda_joint3', u'panda_joint4', u'panda_joint5', u'panda_joint6', u'panda_joint7', u'panda_joint8', u'panda_hand_joint', u'panda_finger_joint1', u'panda_finger_joint2', u'realsense_camera_joint', u'realsense_optical_joint', u'realsense_rgb2depth_virtual_joint', u'realsense_fake_joint', u'panda_hand_to_tool', u'panda_ee_link_joint', u'world_link', u'table_joint', u'kinect_camera_joint', u'kinect_opengl_virtual_joint', u'kinect_rgb2depth_virtual_joint'])
('joint indexes:         ', 22, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21])
('actuated joint names:  ', 9, [u'panda_joint1', u'panda_joint2', u'panda_joint3', u'panda_joint4', u'panda_joint5', u'panda_joint6', u'panda_joint7', u'panda_finger_joint1', u'panda_finger_joint2'])
('actuated joint indexes:', 9, [0, 1, 2, 3, 4, 5, 6, 9, 10])
('link names:            ', 22, [u'panda_link1', u'panda_link2', u'panda_link3', u'panda_link4', u'panda_link5', u'panda_link6', u'panda_link7', u'panda_link8', u'panda_hand', u'panda_leftfinger', u'panda_rightfinger', u'realsense_link', u'realsense_color_optical_frame', u'realsense_depth_optical_frame', u'realsense_camera_frame', u'tool', u'panda_ee_link', u'world', u'table', u'opengl_rgb_camera_link', u'rgb_camera_link', u'depth_camera_link'])
('link indexes:          ', 22, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21])
('joint dampings:        ', [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
('joint frictions:       ', [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
('positionContwrol', 'enabled!')
panda_leftfinger friction updated!
panda_rightfinger friction updated!
[INFO] [1634625989.617703]: Robot loaded
('robot id:', 2)
Box friction updated!
('lateralFriction:', 1.0)
('spinningFriction:', 1.0)
Box friction anchor enabled!
kinect base position:[0.916 0.006 1.472]
kinect base rpy:(0.015193733022188204, 0.5349253756133845, -3.1338467992197665)
b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
camera_bodyb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
camera_baseb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
No inertial data for link, using mass=1, localinertiadiagonal = 1,1,1, identity local inertial frameb3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:
camera_visor****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
('robot name:            ', u'azure-kinect')
('robot ID:              ', 3L)
****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
('robot ID:              ', 1)
('robot name:            ', u'panda')
('robot total mass:      ', 17.072002999999995)
('base link name:        ', u'panda_link0')
('num of joints:         ', 22)
('num of actuated joints:', 9)
('joint names:           ', 9, ['panda_joint1', 'panda_joint2', 'panda_joint3', 'panda_joint4', 'panda_joint5', 'panda_joint6', 'panda_joint7', 'panda_finger_joint1', 'panda_finger_joint2'])
('joint indexes:         ', 22, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21])
('actuated joint names:  ', 9, [u'panda_joint1', u'panda_joint2', u'panda_joint3', u'panda_joint4', u'panda_joint5', u'panda_joint6', u'panda_joint7', u'panda_finger_joint1', u'panda_finger_joint2'])
('actuated joint indexes:', 9, [0, 1, 2, 3, 4, 5, 6, 9, 10])
('link names:            ', 22, [u'panda_link1', u'panda_link2', u'panda_link3', u'panda_link4', u'panda_link5', u'panda_link6', u'panda_link7', u'panda_link8', u'panda_hand', u'panda_leftfinger', u'panda_rightfinger', u'realsense_link', u'realsense_color_optical_frame', u'realsense_depth_optical_frame', u'realsense_camera_frame', u'tool', u'panda_ee_link', u'world', u'table', u'opengl_rgb_camera_link', u'rgb_camera_link', u'depth_camera_link'])
('link indexes:          ', 22, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21])
('joint dampings:        ', [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
('joint frictions:       ', [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0])
****************************************************************************************************
PyBullet Robot Info 
****************************************************************************************************
tf_broadcaster_initialized!
(0, 'plane', ((0.0, 0.0, 0.0), (0.0, 0.0, 0.0, 1.0)))
(1, 'panda', ((-0.42, 0.0, 0.78), (0.0, 0.0, 0.0, 1.0)))
(2, 'orion_pie_v1', ((0.0, 0.0, 0.14), (0.0, 0.0, 0.0, 1.0)))
reset pose of orion_pie_v1
[INFO] [1634626056.978997]: /position_joint_trajectory_controller/follow_joint_trajectory: Executing
^C[position_joint_trajectory_controller_spawner-5] killing on exit
[joint_state_controller_spawner-4] killing on exit
[robot_state_publisher-3] killing on exit
[ros_interface-2] killing on exit
[pybullet_env-1] killing on exit
[pybullet_env-1] escalating to SIGTERM
[ros_interface-2] escalating to SIGTERM
[pybullet_env-1] escalating to SIGKILL
[ros_interface-2] escalating to SIGKILL
Shutdown errors:
 * process[ros_interface-2, pid 1979]: required SIGKILL. May still be running.
 * process[pybullet_env-1, pid 1978]: required SIGKILL. May still be running.
shutting down processing monitor...
... shutting down processing monitor complete
done
root@marley:/# 

rosbag info 0-0try3.bag OUTPUT

root@marley:~/upload# rosbag info 0-0try3.bag
path:        0-0try3.bag
version:     2.0
duration:    9:37s (577s)
start:       Oct 19 2021 11:12:54.51 (1634613174.51)
end:         Oct 19 2021 11:22:31.68 (1634613751.68)
size:        68.0 KB
messages:    23
compression: none [1/1 chunks]
types:       control_msgs/FollowJointTrajectoryActionGoal [cff5c1d533bf2f82dd0138d57f4304bb]
             control_msgs/GripperCommandActionGoal        [aa581f648a35ed681db2ec0bf7a82bea]
topics:      /franka_gripper/gripper_action/goal                                   3 msgs    : control_msgs/GripperCommandActionGoal
             /position_joint_trajectory_controller/follow_joint_trajectory/goal   20 msgs    : control_msgs/FollowJointTrajectoryActionGoal

Note: The bag file recorded the trajectories followed by the agent when GUI was on. When played with GUI on, it works fine, it, however, fails when GUI is set to off via the above 5 steps

[GouMinghao]

Thanks for your question, we will look into the problem and give you the solution as soon as possible.