SO-101 tabletop (Isaac Sim)
A LeRobot EnvHub environment built natively in NVIDIA Isaac Sim 6.2: an SO-101 arm in the turquoise of the LeRobot kits, four painted wooden 3 cm cubes and a green open box on a table in a simple room, simulated by PhysX and rendered with RTX.
The layout, cameras, observations, actions and LEROBOT_SIM match the MuJoCo version, lerobot/so101-tabletop-mujoco, so the same policies and commands run against both.
| Observation | pixels: front (scene camera) and wrist (gripper camera), 480×640 RGB; agent_pos: 6 joint positions |
| Action | 6 absolute joint position targets: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll in degrees, gripper 0-100 (the units of LeRobot's so101_follower with use_degrees=True) |
| Control rate | 30 fps (8 PhysX steps per control step) |
| Task | "Put all the cubes in the green box." |
| Reward | the fraction of cubes in the box at each step (0, 0.25, ... 1); info["cubes_in_box"] gives the count |
| Episode end | success (terminated) when all four cubes are in the box; truncated after 3600 steps (2 minutes), set with episode_length |
| Reset | seeded cube positions and yaws, joint noise |
Isaac Sim starts once per process, in make_env: about 20 s, plus a few minutes of shader compilation the first time on a machine or in a container. It holds one world, so make_env takes n_envs=1. A step takes about 80 ms on an RTX 3080 Laptop GPU, so lockstep evaluation is exact, while realtime runs slower than 30 fps. The arm asset is bundled, so the environment needs no network access at runtime.
With the LeRobot simulator server
From a LeRobot checkout, start the server. The image installs packages.txt (system libraries) and requirements.txt (Isaac Sim 6.2 from PyPI and NVIDIA's index, Python 3.12); it needs an NVIDIA RTX GPU and the NVIDIA Container Toolkit:
lerobot-env-server --docker --envhub=lerobot/so101-tabletop-isaac
Then drive it as a robot, here with a policy:
lerobot-rollout --robot.type=sim --robot.endpoint=tcp/127.0.0.1:7448 \
--robot.env.type=envhub --robot.env.hub_path=lerobot/so101-tabletop-isaac --robot.env.task=so101_tabletop \
--policy.path=<your SO-101 checkpoint> --fps=30
Without --robot.endpoint, the same command runs the environment in-process (with --robot.env.trust_remote_code=true and the requirements installed next to LeRobot). lerobot-eval takes the same options as --env.*, e.g. --env.type=envhub --env.hub_path=lerobot/so101-tabletop-isaac --env.task=so101_tabletop.
Observations reach policies as observation.images.front, observation.images.wrist and observation.state; use --rename_map if your checkpoint names its cameras differently.
In-process
Install Isaac Sim 6.2 next to LeRobot (uv pip install -r requirements.txt with UV_INDEX_STRATEGY=unsafe-best-match, or pip install -r requirements.txt), then:
from lerobot.envs.factory import make_env
envs = make_env("lerobot/so101-tabletop-isaac", n_envs=1, trust_remote_code=True)
How the scene is built
env.py is a compact example of the Isaac Sim 6 Python API:
| Part | Isaac Sim API |
|---|---|
| Arm | NVIDIA's RobotStudio/so101_new_calib asset (bundled in so101_new_calib/), referenced with stage_utils.add_reference_to_stage, driven through Articulation |
| Room, table, cubes, box, lights | GroundPlane, scaled Cube prims, DomeLight, DistantLight; GeomPrim colliders, RigidPrim bodies, RigidBodyMaterial |
| Looks | OmniPbrMaterial for the wooden cubes (textures/wood.png tinted per cube, with projected UVs since USD cubes have none) and for the arm's color (an override of the asset's own material), PreviewSurfaceMaterial for the table and the glossy box; colors are the MuJoCo scene's sRGB values, converted to Isaac's linear space |
| Cameras | RtxCamera + CameraSensor (RGB annotator); the wrist camera is parented to the arm's gripper link |
| Stepping | timeline paused, SimulationManager.step for physics, a Replicator capture step for the cameras |
Licenses
- This environment is Apache-2.0, including
textures/wood.png(generated procedurally for it). so101_new_calib/is NVIDIA's SO-101 asset from the Isaac Sim 6.2 asset library (Isaac/Robots/RobotStudio/so101_new_calib, thumbnails left out), unchanged (env.pyoverrides its color on the stage), under the Apache-2.0 license in that folder.- Isaac Sim is distributed under the NVIDIA Isaac Sim license.
env.pysetsOMNI_KIT_ACCEPT_EULA=YES; running this environment means you accept it.