GR00T-N1.7 SO101 Left Arm, 11-task

NVIDIA Isaac GR00T N1.7 3B finetuned on the 11 datasets of the SO101_LeftArmOnly_Task collection, merged into one multitask set. One model covers all 11 tasks; the language instruction selects the behaviour.

Trained with the same batch size and step count as twanghcmut/pi05-so101-left-arm-multitask, on the same data.

Tasks

Use the exact wording, including trailing punctuation; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	`Pick up the red cube and place it into the steel pot.`	50
1	`Pick up the blue cube and place it into the steel pot`	50
2	`Take the banana and place it into the steel pot then close the lid.`	50
3	`Pick up the blue cube and place it next to the blue cube.`	50
4	`Take the blue cube out of the steel pot and place it on the blue cube.`	51
5	`Take the red cube out of the steel pot and place it on the red cube`	50
6	`Stack the yellow cube on top of the orange cube.`	50
7	`Pick up one orange block and stack it neatly on top of the other orange block`	50
8	`Pick up the cube that makes the number of red and blue cubes on the wooden board equal, and place it in the empty corner of the wooden board.`	50
9	`Pick up the red cube from the wooden board, put it into the orange hollow cube, then pick up the orange hollow cube and place it on the wooden board`	50
10	`Take out the blue cubes from shortest to tallest and place it on the table`	50

551 episodes, 253,745 frames in total.

Embodiment

SO101 follower arm (left), 5 joints plus gripper.


Embodiment tag	`NEW_EMBODIMENT`
State / action dim	6: `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll` + `gripper`
Cameras	`top` (overhead, 360×640), `wrist` (480×640) RGB
Action horizon	16
Data fps	30

Action representation: single_arm is RELATIVE (delta from current state), gripper is ABSOLUTE. The policy server unnormalizes both, so get_action returns absolute joint targets either way.

Observation format

Nested, as Gr00tPolicy expects. Do not start the server with --use-sim-policy-wrapper; that switches it to the flat format used by the sim envs.

obs = {
    "video": {                       # (B, T, H, W, C) uint8, T == 1
        "top":   top_rgb[None][None],
        "wrist": wrist_rgb[None][None],
    },
    "state": {                       # (B, T, D) float32, T == 1
        "single_arm": joints[:5].reshape(1, 1, 5).astype("float32"),
        "gripper":    joints[5:6].reshape(1, 1, 1).astype("float32"),
    },
    "language": {"annotation.human.task_description": [[INSTRUCTION]]},
}
action, info = client.get_action(obs)   # action["single_arm"] (1,16,5), action["gripper"] (1,16,1)

Serving

python gr00t/eval/run_gr00t_server.py \
  --model-path twanghcmut/gr00t-n1d7-so101-left-arm-multitask \
  --embodiment-tag NEW_EMBODIMENT --host 0.0.0.0 --port 5555

Training


Base	`nvidia/GR00T-N1.7-3B`: projector, VL layernorm and diffusion head trained; LLM and vision backbone frozen
Steps	6,000
Global batch size	128 (768k samples)
Learning rate	1e-4, cosine, 5% warmup
Weight decay	1e-5
`episode_sampling_rate`	0.1
Hardware	1× H100 80GB
Runtime	9.2 h (dataloader-bound, 23 samples/s)
Final train loss	0.0256 (mean over steps 5,760–6,000)

Loss at steps 500 / 1,000 / 2,000 / 3,000 / 4,000 / 5,000 / 6,000: 0.075 → 0.054 → 0.049 → 0.035 → 0.030 → 0.027 → 0.026.

Open-loop evaluation

One trajectory per task, up to 400 steps, action horizon 16, run against the training data with gr00t/eval/open_loop_eval.py. Errors are unnormalized, in the dataset's own action units.

traj	task	MAE
0	0 red cube → pot	2.098
50	1 blue cube → pot	2.327
100	2 banana → pot, close lid	3.258
150	3 blue cube next to blue cube	3.372
200	4 blue cube out of pot → blue cube	2.374
251	5 red cube out of pot → red cube	2.941
301	6 stack yellow on orange	2.604
351	7 stack orange blocks	3.567
401	8 balance red/blue on board	3.717
451	9 red cube via hollow cube	2.760
501	10 blue cubes shortest → tallest	3.385
	average	2.946 (MSE 25.02)

Divided by the absolute action std (mean 28.67 across dims) that is 10.27% of action scale at horizon 16. The π₀.₅ checkpoint trained on the same data and budget scores 9.82% on the same 11 trajectories; GR00T is lower-error on 5 of them (one, traj 100, by only 0.001). The two numbers come from each framework's own open-loop script, so treat a gap this small as a tie rather than a ranking.

Limitations

No held-out split. All 551 episodes were used for training. episode_sampling_rate subsamples timesteps within episodes, not episodes themselves, so nothing was withheld. The open-loop numbers above are therefore measured on training data: they show the policy fits its data and is not degenerate, but they do not measure generalization, and they are not a success rate. Treat real-robot performance as unmeasured until you run it.

One trajectory per task is a small sample; per-task numbers above are single episodes.

Deployment timing

The training data is 30 fps. A client that interpolates between model steps must derive its sub-step count from that ratio: smooth_step = control_hz / 30.