🐛 Segfault when launching node in a container

[vinnnyr]

I will work on getting further details when I can create a more minimal example, but on an ubuntu 20.04 container I am consistently getting segfaults after the world is loaded.

Environment:

Ubuntu 20.04 container

Version / Installation:

Apt install via ROS (0.12.)

root@pop-os:/opt/overlay_ws# apt show ros-humble-mvsim

Package: ros-humble-mvsim
Version: 0.12.0-1jammy.20241209.153119
Priority: optional
Section: misc
Maintainer: Jose-Luis Blanco-Claraco <[email protected]>
Installed-Size: 13.2 MB
Depends: libc6 (>= 2.34), libgcc-s1 (>= 3.3.1), libprotobuf23 (>= 3.12.4), libstdc++6 (>= 11), libzmq5 (>= 4.0.1+dfsg), ros-humble-mrpt-libbase, ros-humble-mrpt-libgui, ros-humble-mrpt-libmaps, ros-humble-mrpt-libmath, ros-humble-mrpt-libobs, ros-humble-mrpt-libopengl, ros-humble-mrpt-libposes, ros-humble-mrpt-libros-bridge, libboost-all-dev, libprotobuf-dev, libprotoc-dev, libzmq3-dev, protobuf-compiler, pybind11-dev, python3-pip, python3-protobuf, python3-venv, ros-humble-mrpt-libtclap, ros-humble-nav-msgs, ros-humble-ros2launch, ros-humble-sensor-msgs, ros-humble-stereo-msgs, ros-humble-tf2, ros-humble-tf2-geometry-msgs, ros-humble-visualization-msgs, unzip, wget, ros-humble-ros-workspace
Homepage: https://wiki.ros.org/mvsim
Download-Size: 3034 kB
APT-Manual-Installed: yes
APT-Sources: http://packages.ros.org/ros2/ubuntu jammy/main amd64 Packages
Description: A lightweight multivehicle simulation framework.

When trying to get the stack trace

Thread 2 "mvsim_node" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 84.112]
0x0000705145205391 in mvsim::World::internalUpdate3DSceneObjects(mrpt::opengl::Scene&, mrpt::opengl::Scene&) () from target:/opt/ros/humble/lib/x86_64-linux-gnu/libmvsim-simulator.so.0.12.0

If I had to guess, it is probably related to it being run inside a container as this works fine on my host.

[jlblancoc]

Does it also happen when invoking in headless mode?

[vinnnyr]

Interestingly enough, the segfault does not happen when using MVSIM CLI inside the container
mvsim launch /opt/ros/humble/share/mvsim/mvsim_tutorial/demo_warehouse.world.xml --headless

But it happens when using the ROS launch
This is the snippet of the launch file

    headless_launch_arg = DeclareLaunchArgument("headless", default_value="True")
    mvsim_node = Node(
        package="mvsim",
        executable="mvsim_node",
        name="mvsim",
        output="screen",
        parameters=[
            {
                "world_file": LaunchConfiguration("world_file"),
                "headless": LaunchConfiguration("headless"),
                "do_fake_localization": False,
            },
        ],
        remappings=[
            ("/tf", "/tf_sim"),
            ("/tf_static", "/tf_static_sim"),
            ("/cmd_vel", "/cmd_vel_sim"),
            ("/lidar1_points", "/center_lidar/all_points"),
        ],
    )

So now I can narrow it down to maybe a few things:

1. Maybe there is something wrong with my world file? (even though it works fine on my host)
2. Maybe the headless parameter is not correctly being given to ROS?

I did notice that in the demo world, using MVSIM CLI, I actually get a readable exception (not a segfault) when trying to run incorrectly with graphics display in the container:

[16:08:58.2243|INFO |mvsim::World] Physics simulation timestep automatically determined as: 5.00 ms
GLFW error 65544: X11: The DISPLAY environment variable is missing
[16:08:58.2319|ERROR|mvsim::World] [internal_GUI_init] Exception: ==== non-MRPT exception ====
Could not initialize GLFW!
[16:08:58.2344|WARN |mvsim::World] [World::getJoystickState()] No Joystick found, disabling joystick-based controllers.
[mvsim_server_thread_update_GUI] Exception: ==== MRPT exception ====
Message:  Timeout waiting for GUI to open!
Location: ./modules/simulator/src/World_gui.cpp:1032: [void mvsim::World::update_GUI(mvsim::World::TUpdateGUIParams*)
Call stack backtrace:
[0 ]     0x7e647954694c 
[1 ]     0x6138bce7fc4d 
[2 ]     0x7e6478cb4253 
[3 ]     0x7e647893aac3 
[4 ] (unknown file) clone

[16:09:23.5398|WARN |mvsim::World] Timeout waiting for async sensors to be simulated in opengl thread.
[16:09:45.8050|WARN |mvsim::World] Timeout waiting for async sensors to be simulated in opengl thread.

This is the behavior I would expect if ROS wasn't being given the headless argument correctly

So now I think the next step might be to compile MVSIM with debug symbols and see what exactly is breaking

[jlblancoc]

You can also give it a try to this docker file example: https://github.com/MRPT/mvsim_docker_example
and if it works, try changing it so it uses ROS 1 noetic instead and check if the problem only affects ROS 1 for some reason (?).

[vinnnyr]

After compiling with debug options I have received some more information for where the segfault is occurring

[gdb-1] Thread 29 "mvsim_node" received signal SIGSEGV, Segmentation fault.
[gdb-1] [Switching to Thread 0x71dcd3e00640 (LWP 2568)]
[gdb-1] mvsim::World::internalUpdate3DSceneObjects (this=0x71dd46cd2020, viz=..., physical=...) at /opt/overlay_ws/src/mvsim/modules/simulator/src/World_gui.cpp:993
[gdb-1] 993                             gui_.gui_win->camera().setCameraPointing(p.x(), p.y(), p.z());

Which seems to relate to the follow_vehicle tag in my world.xml

mvsim/modules/simulator/src/World_gui.cpp

Line 985 in 8cff292

// Camera follow modes:

I removed that item from my world XML, will confirm here if that completely resolves the problem.