minodori/omx_tomato_act

来源：🤗🏴‍☠️ 救援者：isiliconroboticsapache-2.052M 参数207 MBsafetensors✓ 6 个校验和5 天前更新

omx_tomato_act

ACT (Action Chunking with Transformers) policy trained with LeRobot 0.6.1. A ROBOTIS OMX follower arm picks up a tomato and puts it into a bin, using two USB cameras (top and wrist).

OMX 팔로워로 토마토를 집어 통에 넣는 ACT 정책이다. 수업 실습용 소규모 모델이다.

Summary

Item	Value
Policy	ACT, resnet18 backbone, chunk_size 100, VAE on
Training data	minodori/omx_tomato (50 episodes, 34,432 frames, 30 fps)
Training	30,000 steps, batch size 8, seed 1000
Inputs	`observation.state` (6 joints), `observation.images.top`, `observation.images.wrist` (640x480, 30 fps)
Output	`action` (6 joints: pan, lift, elbow, wrist_flex, wrist_roll, gripper)
Hardware	OMX leader (teleoperation) and OMX follower, 2 USB cameras

The tomato was placed at 5 positions during recording (10 episodes each).

Usage

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("minodori/omx_tomato_act")

For real-robot rollout, enable temporal ensembling. The saved config has it off (chunk_size 100 = 3.3 s at 30 fps), and without it the arm jumps at every chunk boundary. Pass these options to the rollout command:

--policy.temporal_ensemble_coeff=0.01 --policy.n_action_steps=1

Temporal ensembling runs inference every step, so it needs a GPU (about 15 ms per inference on an RTX 5060 Laptop; about 200 ms on CPU, too slow for a 30 fps loop).

Limitations (what we saw on the real robot)

This model is sensitive to how the deployment setup differs from the recording.

Top camera position. A few pixels of shift changed where the gripper went. We aligned the live view over a recorded first frame before each test.
Start pose. Starting from a hand-placed home pose that differed from the demonstrations caused a swing of about 5 degrees at the start. Match the recorded start pose (mean of the 50 demos: pan 2.4, lift -63.7, elbow 55.7, wrist_flex 45.0, wrist_roll 1.6, gripper 50.6).
Background in the top image. A hand in view, the cable route, and the background on the right side shifted the target to the right by 10 to 30 mm.
Position results (final settings: step 30000, temporal ensembling on, start pose matched, camera aligned, background and chair position matched to the recording, hands out of view). Small trial counts; treat as anecdotal.

Position	Result
1 (center)	gripper lands slightly right of the tomato (under 20 mm, not measured precisely); the wide-open gripper still grasps it in most trials; 4 of 5 grasps succeeded
3	about 10 mm right; succeeded
4	about 10 mm right; 3 of 3 succeeded
5, 6	almost no offset; succeeded
untrained points between 1-3 and 1-6	succeeded

Not tested: other objects, other tables, other lighting, a different camera mount. Wrist roll barely varies in the training data, so rotated objects will likely fail.

Demo videos

Real-robot rollouts with the final settings (1080p, about 20 seconds each, filmed by hand, so the view moves):

The files are in the assets/ folder of this repository. They are HEVC (H.265) mp4 files. Some browsers cannot play HEVC; if the link does not play, download the file and open it with a video player.

Reproduce

Train with the dataset above using lerobot-train (policy type act, 30,000 steps, batch size 8). The scripts used for recording, training, and rollout are in the GitHub repository minodori/omx-tomato-act.