SyncWorld multi-view (hero_wrist, 512x576), step 56000

A SyncWorld forward-dynamics (FDM) world model on Cosmos3-Nano that sees three cameras as one 512x576 canvas per timestep: the wrist camera large on top (512x384) and the two third-person cameras below it (256x192 each), as in DROID's concat_view. Given actions, a calibration clip (12 segments x 5 frames, interval 3) and a sparse history (up to 25 frames, stride 3), it generates the 16 frames after the current one. It is a weights-only fork of the single-camera run gh-cm-ax09-calib12-ph075-p7sc0z72 at 174,000. Its siblings differ only in the canvas: hero512-step52000 and strip384-step22000.

MAST run gripperhead-mv-herowrist512-4x3-fork174k-4x8: jobs gh-mv-herowrist-qms5lfsk and gh-mv-herowrist-c01d26dk (rl_research, 2026-09-28), then gh-mv-herowrist-q4ckp374 (spatial_ai_research, 2026-10-04, stopped 2026-10-12 at 76,880).

Training

• Initialization: weights of gh-cm-ax09-calib12-ph075-p7sc0z72 at 174,000, a run that saw one third-person camera. Optimizer, scheduler and trainer state are fresh, so steps restart at 0.

• Data: sampled uniformly over the merged episodes of four shard sets (digests in training/manifest.json):

  shard set	camera setups	episodes
  calib_datasets_shards_robocasa_v2	12,242	59,391 (47,154 counterfactual)
  calib_datasets_shards_robomimic_v2	16,608	109,739 (93,131 counterfactual)
  calib_datasets_shards_rlbench_wrist_mv	20,440	58,104 (38,244 perturb)
  droid_fdm_2cam_wide_shards/train	69,708	69,708 (expert only; 910x512 videos)

• Recipe: actions are per-frame end-effector deltas (backward_framewise, euler_xyz, cm/deg) plus the gripper, with no normalization. Calibration is included with p 0.9 (otherwise 12 black null items and a forced history of at least 9 real frames), history with p 0.75. The future has 5-16 real frames, tail-padded to 16, and its action is nulled with p 0.1 for action CFG. Augmentation: axis relabeling (p 0.9), photometric jitter and the multi-view plan below. AdamW at a constant LR of 5e-5 (5x on the action layers; weight decay 0.05) trains the generation expert, the time embedder, the VAE/action projections and the action modality embedding; shift 5, vision loss only. 4 nodes x 8 H100, 32-way FSDP, 4 samples per rank, bf16 parameters (FP32 gradient reduction), EMA off. Config: config.yaml; job env and overrides: training/launch.txt.

• Why 56000: the best of the evaluated 32k/42k/56k/72k, with the highest canvas PSNR and SSIM and the highest third-person PSNR. At 72k the canvas is flat (-0.03 dB) and tp1 is lower in 22 of 25 episodes, so the run was stopped.

Multi-view tiling

Code: cosmos_framework/data/vfm/action/view_tiling.py (geometry, per-sample plan) and GripperheadFDMDataset._compose_multiview (pixels). The eval uses the same code, so its canvas is byte-equal to an is_train=False training item. tiling/tiling.json is generated from that code and the job's env.

• Roles: tp0 is the main third-person camera (first of agentview_rgb, front_rgb, ...), tp1 its <main>_1 sibling, wrist wrist_rgb / eye_in_hand_rgb / hand_rgb. Unit cell: 256x192 (4:3).
• Crops: each view is cropped to 4:3, then given an antialiased resize. 16:9 sources (910x512: official RoboLab, DROID) keep the central 682x512 (x 114-796). Square 512x512 sources keep 384 rows starting at int(128 x crop_y). At eval that is rows 64-448 for third-person views and 128-512 for the wrist, which keeps the gripper (at the bottom) and fills its slot without resampling.
• Training: one plan per sample. Third-person cameras shuffle over slots 0 and 1; the wrist stays in slot 2. One view is blacked out with p 0.10 and two with p 0.05, so each camera (and the large cell) is black in 6.7% of samples. crop_y is drawn per camera, photometric jitter is applied per tile, and missing cameras leave black slots.
• Eval: the canonical plan puts tp0, tp1 and wrist in slots 0, 1 and 2, drops nothing and fixes crop_y at 0.5 / 1.0. An episode whose leaf or calibration clip lacks a camera is skipped.
• Shared plan: the plan is drawn from the episode's cameras before any item is built. It is applied by role to the 12 calibration segments (from the calibration clip's own cameras), the history and the current+future item, so a camera never moves, changes crop or vanishes within a sample. If a calibration clip has none of the shown cameras, it becomes the null calibration.
• Caption: A robot arm interacts with the scene. This video contains concatenated views from multiple camera perspectives. The video is 1.0 seconds long and is of 15 FPS. This video is of 576x512 resolution.
• Latents: tier mv_hero_wrist_256_4x3, 16 x 18 = 288 tokens per frame. The job's MAX_VAE_LATENT_SIDE=32 is not read by the network (unified_3d_mrope).

Eval canvases from _read_eval_episode in examples/eval_gripperhead_fdm_rollout.py, with the flags below. Left: RoboLab mv25 episode_0000 (official 16:9 cameras, BananaInBowlTask, frame 30). Right: episode_0019 (randomized 512x512 cameras, PickOrangeObjectTask, frame 45). Bottom: the raw frames, shaded outside each crop window.

Evaluation on RoboLab mv25

25 RoboLab episodes not in the training data, recorded with Cosmos3-Nano-Policy-DROID (up to 320 frames, 8 successes): 15 with the official 16:9 cameras, 10 with randomized 512x512 orbit cameras, all with calibration. Each segment (one every 16 frames) runs 2 rounds of 16 frames, first on real history, then on its own output. Settings: calibration 12x5@3, history 25@3, action CFG 5, 35 UniPC steps, shift 5, training-matched history padding, caption and CFG null. Means over 421 segments; per-role metrics use a common 256x192 4:3 field of view cut from each slot.

From 42k to 56k, tp0 and tp1 improve in all 25 episodes (by 1.05 and 1.74 dB, mean per-episode gain). Each sibling layout scores best on the camera it shows largest. This one has the best wrist PSNR (22.21, against 21.07 for hero512 at 52k and 21.21 for strip384 at 22k) and the lowest third-person PSNR. Canvas metrics are not comparable across layouts: two thirds of this canvas is the wrist view. With 25 episodes, a few tenths of a dB between checkpoints is noise.

Training losses

Canvas vision flow-matching loss (train@2_detail/flow_matching_loss_vision, equal to train@2/loss: no action loss), mean of the 2,000 updates before each mark (a metrics.jsonl record averages 20 updates over 32 ranks). Training-mix losses, not comparable across layouts.

Files

model/ holds net.* without the frozen Qwen3-VL vision tower (unused by FDM generation), in bf16 exactly as the trainer saved it (the run trained bf16 parameters), with no optimizer or dataloader state. The same DCP is held on Manifold: samworlds_mast/tree/users/yuncongyang/eval_holds/gripperhead-mv-herowrist512-4x3-fork174k-4x8/iter_000056000/model.

Loading (cosmos-framework-dev)

hf download yyuncong/SyncWorld-multiview-herowrist512-step56000 --local-dir ./SyncWorld-multiview-herowrist512-step56000
# base config (docs/training.md, Step 2): python -m cosmos_framework.scripts.convert_model_to_dcp -o examples/checkpoints/Cosmos3-Nano --checkpoint-path Cosmos3-Nano
export WAN_VAE_PATH=/path/to/Wan2.2_VAE.pth      # local Wan2.2 VAE
export EVAL_QWEN_ASSETS=/path/to/qwen3vl_assets   # offline Cosmos3-Nano processor files
export TORCH_HOME=/path/to/torch_home             # LPIPS AlexNet weights (or pass --no-lpips)
PYTHONPATH=. python examples/eval_gripperhead_fdm_rollout.py \
  --checkpoint ./SyncWorld-multiview-herowrist512-step56000 --base-config-dir examples/checkpoints/Cosmos3-Nano \
  --eval-set ./robolab_mv25 --out ./eval_out --tag mv25-herowrist-i56k \
  --num-history-frames 25 --history-frame-stride 3 --num-pred-frames 16 \
  --use-calibration --calib-segments 12 --calib-seg-len 5 --calib-frame-interval 3 \
  --action-convention backward_framewise --action-rot-format euler_xyz \
  --action-trans-scale 100 --action-rot-scale 57.2958 --action-norm none \
  --action-cfg-scale 5.0 --num-steps 35 --fps 15 \
  --multiview-layout hero_wrist --multiview-cell-aspect 4x3 --multiview-expect-tier mv_hero_wrist_256_4x3

The repo dir has no config.json, so the loader builds the model from --base-config-dir (Cosmos3-Nano with model/config.json) and partially loads model/. --multiview-layout sets the other defaults to the training-matched settings above. Results: <out>/<tag>/mv_hero_wrist_256_4x3/; torchrun --nproc_per_node=N shards episodes over GPUs. Eval set: Manifold samworlds_mast/tree/users/yuncongyang/eval_sets/robolab_mv25.