FloMo midtrain base checkpoints
FloMo does any-to-any forward-dynamics cotraining for robot learning from video. This repo has the three midtrain base checkpoints from the paper. Each one is a LoRA on the Wan2.2-TI2V-5B DiT. Code: https://github.com/pairlab/flomo
| Folder | Outputs | Train steps | Size |
|---|---|---|---|
ma/flomo_ma.pt |
motion tokens + action | 13,750 | 646 MB |
va/flomo_va.pt |
video + action | 14,000 | 646 MB |
vma/flomo_vma.pt |
video + motion tokens + action | 13,500 | 646 MB |
Motion tokens are dense 3D scene flow encoded by the frozen Wan VAE. All three checkpoints come from epoch 28 of training on the paper mixture: EgoVerse 0.65, MolmoAct2 0.15, RealYAM 0.15, and RealYAM human video 0.05. Action supervision uses the MolmoAct2 and RealYAM robot data only.
Contents
Each .pt file is a torch.save dict with these keys:
model: trainable parameters (LoRA adapters and FloMo heads). The frozen Wan VAE and umT5 text encoder are loaded from the Wan weights. Inmaandvma, it also holds the per-dataset motion-token scale stats (modalities.motion_tokens.per_ds_*and_extra_state) thatdecode_positionsneeds. Dataset rows:egoverse=0, molmoact2=1, realyam=2, realyam_human=3.lora_enabled:True.normalizer_stats: action normalizers forrealyamandmolmoact2.config: the resolved training config. Absolute paths are set to???, and the W&B ids are removed.
The files do not contain optimizer, scheduler, or scaler state.
track_stats.json lists the same motion-token stats in plain JSON
(0.1 / 99.9 percentiles of per-step velocity and cumulative displacement).
The motion-token stats were added on 2026-11-19 (pairlab/flomo#1). Loading these files needs FloMo code from that date or later. All other tensors are unchanged.
Usage
Install FloMo and the Wan weights as the GitHub README describes. Then:
hf download shitijgovil/FloMo vma/flomo_vma.pt --local-dir ./flomo_ckpts
# Inference: (T, 14) YPR action chunk for a bimanual robot
python infer_example.py --checkpoint ./flomo_ckpts/vma/flomo_vma.pt --wan-ckpt-dir ./wan_checkpoints \
--image primary.png --image left.png --image right.png \
--task "pick up the croissant and place it in the oven" \
--ee-pose="x,y,z,yaw,pitch,roll,grip,x,y,z,yaw,pitch,roll,grip"
# Fine-tune (weights only), e.g. one RoboTwin 2.0 task
python train.py --config-name train_robotwin data_root=<RoboTwin>/data \
robotwin_task=open_laptop checkpoint=./flomo_ckpts/vma/flomo_vma.pt resume=false run_name=robotwin_open_laptop
When you fine-tune VA or VMA on a config with no video output, the loader warns
Unexpected keys: modalities.video_primary.{mask_token,view_pe}. This is expected.
SHA-256
3fe524a8f2eb8dd89ced25bafc206bca42b3cff015120205501cc0012009c4f0 ma/flomo_ma.pt
c50dd53732951f338ed28d34f57aa5f2dabd5da98fe495fc3194b5270655ceaa va/flomo_va.pt
f1b4208c4c9d8da23040468455015d4f9658ec4e3a474784195146ecb646e9cc vma/flomo_vma.pt
175f212a2fa41862935dfef830aeb89f5123b4bb52cad9fe80efb43ad1e231fc track_stats.json
License
The FloMo code and these weights use the MIT license. The base model Wan2.2-TI2V-5B uses the Apache-2.0 license.