SmolVLA distilled with DAgger from MolmoAct2 (LIBERO-10 task 6)

Measured success: 58.0 % on LIBERO-10 task 6 ("put the white mug on the plate and put the chocolate pudding to the right of the plate"), 50 episodes, seed 1000.

Student SmolVLA (0.45B) trained with DAgger (Ross et al., 2011) using MolmoAct2 (5B) as the expert.

Method

Adds the correction step that plain distillation is missing.

1. Collected teacher rollouts (36 successful episodes)
2. Ran the distilled student in the environment and saved every episode it visited, including failures (30 episodes, 12 of them failures)
3. Asked the teacher to label those student-visited states
4. Dataset Aggregation: merged teacher trajectories with the corrected student states
5. Retrained from lerobot/smolvla_base on the union

The failures matter: teacher success trajectories never contain the states where the student actually goes wrong, so a student trained only on them never learns recovery.

Result: +2 pp over plain distillation. Aggregating 2.3x more data, targeted exactly at the student's failure modes, moved 56 % to 58 %.

Evaluation

lerobot-eval, libero_10 task 6, seed 1000, 50 episodes, n_action_steps=10.

MolmoAct2 was previously listed here as 100 %. That number came from a 2-episode sample (2/2) inside a suite-wide run, not from this 50-episode protocol. Re-measured under the same condition as the other rows: 88.0 % (44/50).

Honest summary: this did not work

These checkpoints are negative results, published so the attempt is documented.

• Three different levers were pulled and all plateaued at 58 %
  • changing the data source (teacher rollouts): +8 pp
  • growing the data 2.3x via DAgger: +2 pp
  • growing trainable parameters ~68x (VLM unfrozen): +0 pp
• An off-the-shelf X-VLA (0.9B) scores 92 % on the same task with no distillation at all
• The remaining explanation is architectural. SmolVLA is thinned three ways (half the VLM layers, 64 visual tokens per frame, 0.75x expert width), and none of that is reachable by training

If you want performance on this benchmark, use X-VLA or MolmoAct2 instead. These checkpoints are useful for studying what distillation does and does not fix.

Caveats

• Single task, 50 episodes. 48 / 56 / 58 % differ by a few successful rollouts and are not statistically separable. Treat the ordering as a direction, not a result.
• Teacher ran in bf16 + AMP (14.2 GB VRAM); students in fp32 (2.3 GB).
• Only the teacher's final 7-D action was transferred. No output distribution, no intermediate features, no chain-of-thought. This is the shallowest form of distillation and likely a reason the gains were small.

Reproduce

Scripts, raw measurements and the full write-up: vla_collect.py, collect_teacher.py, collect_student.py, relabel_student.py, merge_datasets.py in the experiment repository.

export MUJOCO_GL=egl PYOPENGL_PLATFORM=egl
lerobot-eval \
  --policy.path=AmberHyunKIM/smolvla_libero10_task6_dagger \
  --policy.device=cuda --policy.n_action_steps=10 \
  --rename_map='{{"observation.images.image":"observation.images.camera1","observation.images.image2":"observation.images.camera2"}}' \
  --env.type=libero --env.task=libero_10 --env.task_ids='[6]' --env.episode_length=600 \
  --eval.batch_size=1 --eval.n_episodes=50 --seed=1000