smolvla-so101-can-butler

A SmolVLA fine-tune that makes a $100 SO-101 arm hand a can to a person: it waits until a human hand appears on the mat, reaches, grasps the can, and delivers it.

Trained on 26 teleoperated demonstrations (plus 16 "stay still" negatives) from espejelomar/so101-can-butler, for 20k steps on a rented RTX 4090 — about $2.30 of compute.

Results

An autonomous grasp by this checkpoint. Left: the overhead camera. Right: the wrist camera — what the policy sees as the jaws close.

3 grasps out of 3 autonomous attempts, each one verified by the servos rather than by eye (see Verification below):

Threshold for "held" is 1.5; failed grasps on this bench measure between −0.41 and 0.52. For honest context: this is a small sample — a larger rate campaign is the next step, and the numbers here should be read as "it works" rather than as a success rate.

A same-data ACT baseline reached 1/3 on the same bench with the same judge; a hand-calibrated classical pipeline (camera → arm homography + scripted grasp) reached 0 autonomous grasps, because absolute camera-to-arm calibration on this setup carried 12–70 mm of error depending on where the can sat. An end-to-end policy needs no such calibration, which is the main practical finding.

Verification (how the grasps were scored)

These servos are position-controlled: the gripper reports the angle it was told to reach, so gripper position alone cannot distinguish a real grasp from closing on air. What can: a can physically prevents the motor from reaching its commanded angle.

Score = median(achieved − commanded) over the hold. Across 26 human demonstrations with a can genuinely held, this gap never fell below 1.84 (median 7.57). Across 22 failed robot attempts it stayed between −0.41 and 0.52. No false positives were observed. Every result above was scored this way, from the recorded episode, with no human labelling.

Usage

The checkpoint expects SmolVLA's own camera names, so map this bench's cameras when running it:

python -m lerobot.scripts.lerobot_rollout \
  --policy.path=espejelomar/smolvla-so101-can-butler \
  --policy.n_action_steps=50 \
  --rename_map='{"observation.images.top": "observation.images.camera1",
                 "observation.images.wrist": "observation.images.camera2"}' \
  --robot.type=so101_follower --robot.port=/dev/ttyACM0 \
  --fps=15 --dataset.single_task="Hand me the can"

Notes:

• n_action_steps must not exceed the model's chunk_size (50).
• Inference measured at ~0.8 s per 50-step chunk on Apple Silicon MPS, i.e. comfortably real time for a 15 Hz loop with full-chunk execution.
• The policy is trigger-conditioned: it deliberately stays still until a human hand enters the frame, because 16 training episodes teach exactly that. An empty scene produces no motion — that is correct behavior, not a failure.

Training

Limitations

• Trained on one object (a foil-wrapped can) in one zone of the workspace; generalization to other objects, lighting, and positions is untested.
• Requires a human hand entering the frame to trigger; it is not an autonomous picker.
• The handover release is a spoken countdown, not force-sensed: these servos show zero measurable load change under a maximum human pull (high gear reduction attenuates it), so pull detection is not physically available on this hardware.
• Small evaluation sample (3 attempts).