GR00T-N1.7-SO101-Multitask

NVIDIA Isaac GR00T N1.7 3B finetuned on hungho77/so101-multitask — one model covering all three tasks, not three separate checkpoints.

Tasks

The language instruction selects the behaviour. Use the exact wording below; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	`Pick up the banana and place it in the bot, then close the lid`	50
1	`Pick blue cube and place on red cube`	49
2	`Pick all cubes and place into cup`	44

Embodiment

SO101 follower arm, 5 joints plus gripper.


Embodiment tag	`NEW_EMBODIMENT`
State / action dim	6 — `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll` + `gripper`
Cameras	`top` (overhead), `wrist` — 480×640 RGB
Action horizon	16
Data fps	30

Action representation: single_arm is RELATIVE (delta from current state), gripper is ABSOLUTE. The policy server unnormalizes both, so get_action returns absolute joint targets either way.

Observation format

Nested, as Gr00tPolicy expects. Do not start the server with --use-sim-policy-wrapper — that switches it to the flat format used by the sim envs.

obs = {
    "video": {                       # (B, T, H, W, C) uint8, T == 1
        "top":   top_rgb[None][None],
        "wrist": wrist_rgb[None][None],
    },
    "state": {                       # (B, T, D) float32, T == 1
        "single_arm": joints[:5].reshape(1, 1, 5).astype("float32"),
        "gripper":    joints[5:6].reshape(1, 1, 1).astype("float32"),
    },
    "language": {"annotation.human.task_description": [[INSTRUCTION]]},
}
action, info = client.get_action(obs)   # action["single_arm"] (1,16,5), action["gripper"] (1,16,1)

Serving

python gr00t/eval/run_gr00t_server.py \
  --model-path twanghcmut/GR00T-N1.7-SO101-Multitask \
  --embodiment-tag NEW_EMBODIMENT --host 0.0.0.0 --port 5555

Deployment timing

The training data is 30 fps. A client that interpolates between model steps must derive its sub-step count from that ratio:

smooth_step = control_hz / 30

Hard-coding a value taken from a 15 fps robot stretches every trajectory by 2x and the arm creeps through the task without finishing it.

Training


Base	`nvidia/GR00T-N1.7-3B` (3.14 B params, 1.62 B trainable — projector + diffusion head; LLM and vision backbone frozen)
Steps	6000
Global batch size	128
Learning rate	1e-4, cosine, 5% warmup
Weight decay	1e-5
Hardware	1× H100 80GB
Runtime	8.6 h
Final train loss	0.0168

Loss by 500-step block: 0.2711 → 0.0617 → 0.0500 → 0.0426 → 0.0369 → 0.0320 → 0.0286 → 0.0249 → 0.0221 → 0.0195 → 0.0175 → 0.0170. The last block moved only −0.0004, so the run had converged by 6000 steps.

Limitations

No held-out split. All 143 episodes were used for training. episode_sampling_rate subsamples timesteps within episodes, not episodes themselves, so nothing was withheld. The open-loop numbers below are therefore measured on training data: they show the policy fits its data and is not degenerate, but they do not measure generalization, and they are not a success rate. Treat real-robot performance as unmeasured until you run it.

One episode (index 49, task 0) is a 5-frame aborted recording, 0.007% of frames. It was left in; every other episode is at least 223 frames.

wrist_roll barely moves across this dataset (std 0.95 absolute), so expect little controllability on that joint.

Open-loop evaluation

Two trajectories per task, 400 steps, action horizon 16, run against the training data.

traj	task	ckpt-4000 MAE	ckpt-6000 MAE
0	banana	1.804	1.383
25	banana	1.618	1.252
60	blue/red cube	1.728	1.748
80	blue/red cube	1.682	1.222
110	cubes→cup	1.283	0.960
130	cubes→cup	1.420	1.113
	average	1.589	1.280

Raw MAE is in the dataset's own action units, where the per-dimension std averages 22.3, so 1.280 is 5.7% of the action scale. checkpoint-6000 beats checkpoint-4000 on 5 of 6 trajectories (19% lower MAE), which is why it is the checkpoint published here — the final 2000 steps improved the fit rather than only memorizing.

Per task, Pick blue cube and place on red cube is the hardest and Pick all cubes and place into cup the easiest.

Read these together with the Limitations section above: they are measured on training data and are not a success rate.