Best Practices

It is possible to improve different aspects of the simulated environment, such as its physical fidelity, sensor models, and many other components that influence the reality gap.

For most robotic tasks, it is possible to define different observation and action spaces. This choice of interface defines the potential behaviors of the resulting system and can have strong implications on the reality gap.

The system design and implementations strongly influence the transferability of any behavior from simulation to the real world. It is important to consider the reality gap when designing the system, including its hardware, controller implementations, constraints, and software stack.

One of the most common approaches to overcome the reality gap is to expose the policy to a large variation of the system and dynamics parameters during training. As a result, the policy should be robust to different instantiations of these parameters, including the ones representing the real world.

Another important category of methods to overcome the reality gap is to carefully select and curate the training data and exploration strategies used in simulation. These methods focus on generating training data that most closely resemble the target real-world data or data from worst-case behaviors.

In addition to the data a policy is exposed to at training time, the policy's architecture and constraints can play a crucial role in the transfer to the real world. This section describes different ways to structure the policy and regularize it in a way that helps sim-to-real transfer.