SmolVLA fine-tuned for simulated peg-in-hole insertion (SO-101)
SmolVLA fine-tuned to insert a 20 mm peg into a hole with 0.25–2 mm radial clearance, using an
SO-101 arm in a MuJoCo simulation. Built for a robotics take-home assignment.
- Base model:
lerobot/smolvla_base at revision d9f33c94a60fb382c90dea2164c96845bd955e28.
- Training data: 1,339 successful episodes from a scripted demonstrator in MuJoCo, with domain
randomization over physics (clearance, friction, peg mass, grasp pose, hole position, latency) and visuals
(lighting, textures, camera pose, distractor objects). About 14% of episodes include a deliberate
missed alignment followed by a spiral-search recovery.
- Fine-tuning: LeRobot 0.6.1 defaults for SmolVLA. The vision-language model is frozen; the action expert
and state projection are trained. Batch 16, 30,000 steps, learning rate 1e-4, chunk size 50, under an
8 GB GPU memory cap.
- Inputs:
observation.state (5 joint angles in degrees plus gripper 0–100), overhead camera as
camera1 and wrist camera as camera2 (640×480 renders), task "Insert the peg into the hole."
- Outputs: 6 absolute joint position targets at 30 Hz.
- Recommended execution: re-plan every 10 actions (
n_action_steps=10). On tight clearances
(0.25–1.5 mm) this succeeded in 81 of 100 episodes, versus 57% when executing full 50-action chunks.
Simulation-only: trained on rendered images and never tested on a real robot.
