GR00T-N1.7 SO101 Left Arm, 3-task

NVIDIA Isaac GR00T N1.7 3B finetuned on three SO101 left-arm datasets merged into one multitask set. One model covers all three tasks, and the language instruction selects the task.

It uses the same data and sample budget as twanghcmut/pi05-so101-left-arm-task3, and the same GR00T recipe as the 11-task twanghcmut/gr00t-n1d7-so101-left-arm-multitask.

Tasks

Use the exact wording, including trailing punctuation. The model has not been trained on paraphrases.

#	Instruction	Episodes
0	`Take the red cube out of the steel pot and place it into the orange hollow cube.`	51
1	`Use the left gripper to open the top drawer, pick up the red square block, place it inside the top drawer, and then close the drawer.`	57
2	`Open the drawer, pick up the red cube then stack on the blue cube`	51

The set has 159 episodes and 133,763 frames. The sources are LeRobot v3.0. They were converted to v2.1, which is what GR00T's loader reads, and merged without re-encoding, so the frames are unchanged. observation.images.top_depth is not used.

Embodiment

SO101 follower arm (left), 5 joints plus gripper.


Embodiment tag	`NEW_EMBODIMENT`
State / action dim	6: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll` + `gripper`
Cameras	`top` (overhead, 360×640), `wrist` (480×640) RGB
Action horizon	16
Data fps	30

Action representation:

single_arm is RELATIVE: a delta from the current state, normalized with the relative-action statistics in statistics.json.
gripper is ABSOLUTE.

The policy converts both back, so get_action returns absolute joint targets.

Observation format

The observation is nested, as Gr00tPolicy expects. Do not start the server with --use-sim-policy-wrapper, because that switches it to the flat format used by the sim envs.

obs = {
    "video": {                       # (B, T, H, W, C) uint8, T == 1
        "top":   top_rgb[None][None],
        "wrist": wrist_rgb[None][None],
    },
    "state": {                       # (B, T, D) float32, T == 1
        "single_arm": joints[:5].reshape(1, 1, 5).astype("float32"),
        "gripper":    joints[5:6].reshape(1, 1, 1).astype("float32"),
    },
    "language": {"annotation.human.task_description": [[INSTRUCTION]]},
}
action, info = client.get_action(obs)   # action["single_arm"] (1,16,5), action["gripper"] (1,16,1)

Serving

python gr00t/eval/run_gr00t_server.py \
  --model-path twanghcmut/gr00t-n1d7-so101-left-arm-task3 \
  --embodiment-tag NEW_EMBODIMENT --host 0.0.0.0 --port 5555

Training


Base	`nvidia/GR00T-N1.7-3B`. Projector, VL layernorm and diffusion head are trained; the LLM and vision backbone are frozen
Steps	6,000
Global batch size	128 (768k samples, 5.7 epochs over 133,763 frames)
Learning rate	1e-4, cosine, 5% warmup
Weight decay	1e-5
`episode_sampling_rate`	0.1
Hardware	1× H100 80GB
Runtime	9.6 h (dataloader-bound, about 22 samples/s)
Final train loss	0.0199 (mean over steps 5,760–6,000)

Loss at steps 500, 1,000, 2,000, 3,000, 4,000, 5,000 and 6,000: 0.068, 0.054, 0.043, 0.030, 0.027, 0.022, 0.020.

Open-loop evaluation

The evaluation uses two trajectories per task. Each trajectory runs for up to 400 steps with action horizon 16, against the training data, using gr00t/eval/open_loop_eval.py. Errors are unnormalized, in the dataset's own action units.

traj	task	MAE	π₀.₅ (same trajectories)
0	0 red cube: pot → hollow cube	1.406	1.286
25	0 red cube: pot → hollow cube	1.542	1.690
51	1 drawer + red block	1.359	1.239
80	1 drawer + red block	1.601	1.704
108	2 drawer, stack red on blue	1.245	1.345
133	2 drawer, stack red on blue	1.486	1.488
	average	1.440 (MSE 5.47)	1.459 (MSE 5.08)

Divided by the absolute action std (29.97, the mean across dims), that is 4.80% of the action scale at horizon 16. The π₀.₅ checkpoint trained on the same data and budget scores 4.87%. GR00T has lower error on 4 of the 6 trajectories, and π₀.₅ has the lower MSE. The two numbers come from each framework's own open-loop script, so treat the gap as a tie rather than a ranking.

open_loop_traj133.png plots the predicted and recorded actions for trajectory 133.

Limitations

No held-out split. All 159 episodes were used for training. episode_sampling_rate subsamples timesteps within episodes, not the episodes themselves, so nothing was held out. The open-loop numbers above are therefore measured on training data. They show that the policy fits its data and is not degenerate. They do not measure generalization, and they are not a success rate. Treat real-robot performance as unmeasured until you run it.

Two trajectories per task is a small sample.

Deployment timing

The training data is 30 fps. A client that interpolates between model steps must derive its sub-step count from that: smooth_step = control_hz / 30.