SmolVLA distilled from MolmoAct2 (LIBERO-10 task 6)

Measured success: 56.0 % on LIBERO-10 task 6 ("put the white mug on the plate and put the chocolate pudding to the right of the plate"), 50 episodes, seed 1000.

Student SmolVLA (0.45B) trained from lerobot/smolvla_base on rollouts produced by teacher MolmoAct2 (5B), which scores 88.0 % on this task (50 episodes, seed 1000).

Method

Plain policy distillation on teacher rollouts.

1. Ran MolmoAct2 in LIBERO and recorded (observation, teacher action) for every step
2. Kept only successful episodes (36 saved out of 38 attempts, 94.7 %)
3. Trained SmolVLA from scratch on those pairs, 20k steps, batch 8

An earlier variant relabelled human demonstrations with the teacher instead. That was abandoned after measurement: teacher and student actions agreed to within 2.7 %, because the teacher had itself been trained on those same demonstrations. Relabelling a teacher's own training distribution transfers almost nothing.

Evaluation

lerobot-eval, libero_10 task 6, seed 1000, 50 episodes, n_action_steps=10.

MolmoAct2 was previously listed here as 100 %. That number came from a 2-episode sample (2/2) inside a suite-wide run, not from this 50-episode protocol. Re-measured under the same condition as the other rows: 88.0 % (44/50).

Honest summary: this did not work

These checkpoints are negative results, published so the attempt is documented.

• Three different levers were pulled and all plateaued at 58 %
  • changing the data source (teacher rollouts): +8 pp
  • growing the data 2.3x via DAgger: +2 pp
  • growing trainable parameters ~68x (VLM unfrozen): +0 pp
• An off-the-shelf X-VLA (0.9B) scores 92 % on the same task with no distillation at all
• The remaining explanation is architectural. SmolVLA is thinned three ways (half the VLM layers, 64 visual tokens per frame, 0.75x expert width), and none of that is reachable by training

If you want performance on this benchmark, use X-VLA or MolmoAct2 instead. These checkpoints are useful for studying what distillation does and does not fix.

Caveats

• Single task, 50 episodes. 48 / 56 / 58 % differ by a few successful rollouts and are not statistically separable. Treat the ordering as a direction, not a result.
• Teacher ran in bf16 + AMP (14.2 GB VRAM); students in fp32 (2.3 GB).
• Only the teacher's final 7-D action was transferred. No output distribution, no intermediate features, no chain-of-thought. This is the shallowest form of distillation and likely a reason the gains were small.

Reproduce

Scripts, raw measurements and the full write-up: vla_collect.py, collect_teacher.py, collect_student.py, relabel_student.py, merge_datasets.py in the experiment repository.

export MUJOCO_GL=egl PYOPENGL_PLATFORM=egl
lerobot-eval \
  --policy.path=AmberHyunKIM/smolvla_libero10_task6_distilled \
  --policy.device=cuda --policy.n_action_steps=10 \
  --rename_map='{{"observation.images.image":"observation.images.camera1","observation.images.image2":"observation.images.camera2"}}' \
  --env.type=libero --env.task=libero_10 --env.task_ids='[6]' --env.episode_length=600 \
  --eval.batch_size=1 --eval.n_episodes=50 --seed=1000