Different ways to handle inference latency in simulation and on real robot?

[MasterXiong]

Hi,

Thanks for open-sourcing this great benchmark! I wanted to ask a general question about how is model inference latency handled in simulation and on real robot? For example, the model takes observation at time t as its input to predict the action, and due to model inference latency, it may get the predicted action at time t'. In simulation (Simpler), I think we are "freezing" the env and wait until the model finished inference right? So that the action will be executed on the same observation that the model uses as its input. But on real robot, the robot still moves when the model does inference, so the action will be executed on an observation that is different from the one used as model input. Will these two different modes cause domain gap between simulation and real world, especially considering the high inference latency of RT-1 and Octo? Thanks!

[xuanlinli17]

We only step simulation after the model has finished inference, so SIMPLER uses a fixed control frequency (3hz for google robot and 5hz for Bridge). Action will be executed on the same observation that the model uses as its input.

In the real world the control frequency can indeed vary due to latency (and Google Robot and WidowX use non-blocking controller I believe). To handle the latency gap, we perform system identification for simulation (fix-frequency blocking control) controller PD parameters such that it matches real-world (non-blocking) controller behaviors, so it doesn't cause much gap in sim & real behaviors.