SmolVLA: pick R2-D2 and put it in the box (SO-100)

SmolVLA, fine-tuned from lerobot/smolvla_base (vision-language part frozen, action expert trained).

Trained on van-i/r2d2_to_box_bg_20261003_210444: 50 teleoperated episodes on an SO-100 arm with three cameras (left and right overhead, grip on the wrist, 640×480 @ 30 fps). Task: "pick r2d2 and put to box": pick up a small R2-D2 figure from one of five taped start positions and drop it into a cardboard tray. Part of a school robotics test stand built with LeRobot and LeLab. Full write-up, comparison of all four policies and scripts: van-i/so100-imitation-learning-stand.

Training

25,000 steps, batch 8 (~14 epochs), command line, 2 h 57 min on an RTX 3090 (250 W). Final training loss 0.062.

Results on the real arm

10/10 on the trained start positions, 2/4 on held-out positions (H1 2/2, H2 0/2), ~8.4 s per success. With the default synchronous inference the arm pauses briefly every ~2 s while the next 50-action chunk is computed.

All four policies trained on the same dataset, 14 tries each (5 trained positions ×2, held-out H1 between two marks ×2, H2 just outside the marked area ×2):

Camera names: the model expects camera1/2/3. Map the robot's cameras with --rename_map (left → camera1, grip → camera2, right → camera3), as in the command below.

Instruction following: it was fine-tuned on a single instruction, so it effectively ignores the task text. Given a new object and a new instruction, it grabbed the object but still carried it to the box.

Run it

LeRobot 0.6.0, SO-100 / SO-101 follower. Adjust the serial port, calibration id and camera indices to your setup:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=van-i/smolvla_r2d2_to_box_bg \
  --policy.device=cuda \
  --robot.type=so101_follower --robot.port=/dev/ttyACM1 --robot.id=<your-follower-calibration-id> \
  --robot.cameras="{left: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}, grip: {type: opencv, index_or_path: 4, width: 640, height: 480, fps: 30}, right: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}}" \
  --rename_map='{"observation.images.left":"observation.images.camera1","observation.images.grip":"observation.images.camera2","observation.images.right":"observation.images.camera3"}' \
  --task="pick r2d2 and put to box" \
  --duration=60

The policy only works in a scene like the training one: dark matte table, the tray at its spot, similar lighting and camera placement.