SmolVLA with the VLM unfrozen (LIBERO-10 task 6)

Measured success: 58.0 % on LIBERO-10 task 6 ("put the white mug on the plate and put the chocolate pudding to the right of the plate"), 50 episodes, seed 1000.

Same DAgger data as smolvla_libero10_task6_dagger, but with the vision-language backbone unfrozen.

Why this run exists

SmolVLA ships with train_expert_only=True, and in the LeRobot implementation that freezes the entire VLM:

if self.train_expert_only:
    self.vlm.eval()
    for params in self.vlm.parameters():
        params.requires_grad = False

The hypothesis was that the 58 % plateau came from never training the VLM. So this run set train_expert_only=false (vision encoder still frozen) and changed nothing else.

Result: 58.0 %, identical to the frozen-VLM run. Growing trainable parameters ~68x changed nothing, which is the strongest single piece of evidence that the ceiling is architectural rather than a training-scope problem.

Evaluation

lerobot-eval, libero_10 task 6, seed 1000, 50 episodes, n_action_steps=10.

MolmoAct2 was previously listed here as 100 %. That number came from a 2-episode sample (2/2) inside a suite-wide run, not from this 50-episode protocol. Re-measured under the same condition as the other rows: 88.0 % (44/50).

Honest summary: this did not work

These checkpoints are negative results, published so the attempt is documented.

• Three different levers were pulled and all plateaued at 58 %
  • changing the data source (teacher rollouts): +8 pp
  • growing the data 2.3x via DAgger: +2 pp
  • growing trainable parameters ~68x (VLM unfrozen): +0 pp
• An off-the-shelf X-VLA (0.9B) scores 92 % on the same task with no distillation at all
• The remaining explanation is architectural. SmolVLA is thinned three ways (half the VLM layers, 64 visual tokens per frame, 0.75x expert width), and none of that is reachable by training

If you want performance on this benchmark, use X-VLA or MolmoAct2 instead. These checkpoints are useful for studying what distillation does and does not fix.

Caveats

• Single task, 50 episodes. 48 / 56 / 58 % differ by a few successful rollouts and are not statistically separable. Treat the ordering as a direction, not a result.
• Teacher ran in bf16 + AMP (14.2 GB VRAM); students in fp32 (2.3 GB).
• Only the teacher's final 7-D action was transferred. No output distribution, no intermediate features, no chain-of-thought. This is the shallowest form of distillation and likely a reason the gains were small.

Reproduce

Scripts, raw measurements and the full write-up: vla_collect.py, collect_teacher.py, collect_student.py, relabel_student.py, merge_datasets.py in the experiment repository.

export MUJOCO_GL=egl PYOPENGL_PLATFORM=egl
lerobot-eval \
  --policy.path=AmberHyunKIM/smolvla_libero10_task6_fullvlm \
  --policy.device=cuda --policy.n_action_steps=10 \
  --rename_map='{{"observation.images.image":"observation.images.camera1","observation.images.image2":"observation.images.camera2"}}' \
  --env.type=libero --env.task=libero_10 --env.task_ids='[6]' --env.episode_length=600 \
  --eval.batch_size=1 --eval.n_episodes=50 --seed=1000