Gazebo died when object was dropped into the bin

[alex07zzz]

It happens every time when the object is dropped to the bin, Gazebo dies with following infomation:
[gazebo-2] process has died [pid 28267, exit code 134, cmd /opt/ros/kinetic/lib/gazebo_ros/gzserver -e ode --verbose /home/alex/my_gilbreth/src/gilbreth/gilbreth_gazebo/worlds/gilbreth.world __name:=gazebo __log:=/home/alex/.ros/log/c22afa4e-02dd-11e8-a045-142d27f202bb/gazebo-2.log].

[jrgnicho]

OK, that's rather odd. The bin plugin is configured to "dispose" of the object whenever one gets dropped in it. @marip8 would you have any idea about what could be causing this?

[dhood]

I hope you don't mind some external input on this, but from my experience using gazebo for ARIAC, I'd also suspect this and #36 to be caused by deleting the object. Adding a mutex to the plugin to prevent race conditions might help, or reverting d84733e so that the model still exists somewhere in the world.

[jrgnicho]

@dhood I appreciate the input very much. I agree with you, it's very likely that deleting the object is causing this problem. In my latest PR, I instead call the delete_model service (also hosted by a gazebo plugin) from a separate thread in the ObjectDisposal Plugin. This showed a significant improvement, however it still ocassionally crashes; usually after the deleting an object that came into contract with the Vacuum gripper. I suspect that some contact data or gripper joint may hold pointers to objects that are not longer in the world but I need to do more testing to confirm this.

[dhood]

I wouldn't be surprised if that's the case, but I don't have any specific suggestions for where it might be occurring off the top of my head. You are probably aware already that the vacuum gripper plugin will store the collision but I didn't think it accessed it except to attach the joint.

I have seen race conditions if gazebo actions are performed in the wrong thread before (e.g. there was once a race condition if the detach was not done in gazebo's World::Runloop thread in the gripper plugin https://bitbucket.org/osrf/ariac/issues/61): you might be in a similar situation.

Sometimes you can reproduce these race conditions more reliably by increasing the RTF of the world; e.g. in the regression test for the gripper detach, we remove all unnecessary plugins and models from the world and unthrottle the physics update rate so it can go above RTF of 1.0. For that issue it made the segfault occur every time instead of just occasionally. That might help your investigation.

[jrgnicho]

@dhood This is great info, I'll do some more digging around the Gripper code to see if its running into the same type of race condition you've observed.

[dhood]

nw, you might not need to resolve it if you end up switching back to a 'teleporting' strategy, but unthrottling physics is a handy trick for debugging race conditions regardless

[jrgnicho]

Yes, I was gonna work on restoring the teleporting approach first and see if this issue still shows up after that change.

[dhood]

fingers crossed 😄

[jrgnicho]

FYI I opened up an issue in the gazebo issue tracker if any of you are interested in following up on that.