GR00T-N1.6-SO101-Multitask

NVIDIA Isaac GR00T N1.6 3B finetuned on hungho77/so101-multitask — one model covering all three tasks, not three separate checkpoints.

Trained as one of three models on identical data, batch size and step count, so the runs are directly comparable; see Comparison below.

Tasks

The language instruction selects the behaviour. Use the exact wording; the model has not been trained on paraphrases.

#	Instruction	Episodes
0	`Pick up the banana and place it in the bot, then close the lid`	50
1	`Pick blue cube and place on red cube`	49
2	`Pick all cubes and place into cup`	44

Embodiment

SO101 follower arm, 5 joints plus gripper.


Embodiment tag	`NEW_EMBODIMENT`
State / action dim	6 — `shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll` + `gripper`
Cameras	`top` (overhead), `wrist` — 480×640 RGB
Data fps	30

Action representation: single_arm is RELATIVE (delta from current state), gripper is ABSOLUTE. The policy unnormalizes both, so it returns absolute joint targets.

Horizon caveat. The modality config supervises 16 future steps (delta_indices 0–15) while the model config's action_horizon is 50, and nothing derives one from the other. Steps beyond 16 are therefore weakly supervised — use the first 16 and evaluate on those. This mismatch is inherited from NVIDIA's own examples/SO100 config, left as-is.

Observation format

obs = {
    "video": {                       # (B, T, H, W, C) uint8, T == 1
        "top":   top_rgb[None][None],
        "wrist": wrist_rgb[None][None],
    },
    "state": {                       # (B, T, D) float32, T == 1
        "single_arm": joints[:5].reshape(1, 1, 5).astype("float32"),
        "gripper":    joints[5:6].reshape(1, 1, 1).astype("float32"),
    },
    "language": {"annotation.human.task_description": [[INSTRUCTION]]},
}

Files

modality.json and so101_config.py are included: the checkpoint cannot be used without a matching modality config, and this one differs from NVIDIA's examples/SO100 (the third-person camera here is top, an overhead view, not front).

Training


Base	`nvidia/GR00T-N1.6-3B` (3.29 B params, 1.62 B trainable — 49.3%)
Backbone	Eagle-Block2A-2B, `tune_top_llm_layers=4`, LLM and vision frozen
Steps	6000
Global batch size	128 (768k samples = 11.4 epochs over 67,496 frames)
Learning rate	1e-4, cosine, 5% warmup, weight decay 1e-5
Hardware	1× H100 80GB
Runtime	8.58 h at 5.15 s/step
Final train loss	0.0129 (mean over steps 5,751–6,000)

Converged: the final 250-step block moved +0.0007 (i.e. stopped improving) and the LR decayed to 7.59e-12.

Loss by 250-step block: 0.6157 → 0.0688 → 0.0477 → 0.0417 → 0.0364 → 0.0330 → 0.0302 → 0.0280 → 0.0253 → 0.0257 → 0.0229 → 0.0218 → 0.0205 → 0.0198 → 0.0183 → 0.0172 → 0.0161 → 0.0158 → 0.0145 → 0.0138 → 0.0129 → 0.0127 → 0.0122 → 0.0129.

This run was GPU-starved, not compute-bound. nvidia-smi sampled during training showed 0% GPU utilization at 125 W of a 700 W cap. The bottleneck is CPU-side video decoding in the sharded dataloader on a shared 8-core box, not the model. Wall-clock here says nothing about the architecture's cost.

Limitations

No held-out split. All 143 episodes were used for training. episode_sampling_rate subsamples timesteps within episodes, not episodes themselves, so nothing was withheld. The numbers below are measured on training data: they show the policy fits its data and is not degenerate, but they do not measure generalization and are not a success rate. Treat real-robot performance as unmeasured.

One episode (index 49) is a 5-frame aborted recording, 0.007% of frames, left in.

wrist_roll barely moves in this dataset (std 0.95 against 16–37 for the other joints), so expect little controllability on that joint.

Deployment note

The data is 30 fps. A client interpolating between model steps must derive its sub-step count from that: smooth_step = control_hz / 30. Reusing a value from a 15 fps robot stretches every trajectory by 2× and the arm creeps without finishing the task.

Open-loop evaluation

Two trajectories per task, 400 steps, action horizon 16, run against the training data. Errors are in the dataset's own action units.

traj	task	ckpt-5000 MAE	ckpt-6000 MAE
0	banana	1.025	0.988
25	banana	1.070	0.759
60	blue/red cube	0.967	1.034
80	blue/red cube	0.817	1.394
110	cubes→cup	0.899	0.748
130	cubes→cup	0.830	0.820
	average	0.935	0.957

Dividing by the absolute action std (mean 22.33 across dims): 4.19% of action scale for ckpt-5000 and 4.29% for ckpt-6000. Published here is ckpt-6000, the converged final step; ckpt-5000 scored 2.4% lower, which is inside the spread across six trajectories rather than a real difference, and is consistent with the last 250-step loss block ticking up.

Comparison

Three models trained on this same dataset with the same batch size (128) and step count (6000), all scored on the first 16 predicted steps of the same six trajectories:

	GR00T N1.6	GR00T N1.7	π₀.₅ (openpi)
Backbone	Eagle-Block2A-2B	Qwen3-VL	PaliGemma + action expert
Runtime	8.58 h	8.62 h	13.8 h
GPU utilization	0%	6%	100%
Final train loss	0.0129	0.0168	0.00302
Converged	yes	yes	no (LR at 94% of peak)
MAE, horizon 16	0.957 = 4.29%	1.280 = 5.73%	1.005 = 4.50%

N1.6 has the lowest average error, though per-trajectory it and π₀.₅ split the wins 3–3 and N1.7 wins none. Two cautions:

Training loss is not comparable across these three. Different backbones, different action normalization, different starting points (π₀.₅ began at 0.0727 because its base is pretrained; the GR00T runs began above 0.6). Only the MAE row compares.
Every number here is measured on training data — see Limitations.

twanghcmut/GR00T-N1.6-SO101-Multitask