SyncWorld multi-view (hero, 512x576), step 52000

A SyncWorld forward-dynamics world model on Cosmos3-Nano that sees three cameras at once. Every frame it reads or generates is one 512x576 (W x H) hero canvas: a third-person camera in a big 512x384 cell on top, the other third-person camera bottom-left and the wrist camera bottom-right (256x192 each). Given a visual calibration clip (12 segments x 5 frames, interval 3), a sparse history (up to 25 real frames, stride 3) and the actions, it predicts the 16 frames after the current one.

This repo holds iteration 52,000 of gripperhead-mv-hero512-4x3-fork174k-4x8. That run is a weights-only fork of gh-cm-ax09-calib12-ph075-p7sc0z72 at iteration 174,000, which was trained on single 512x512 views. Sibling layouts share the recipe and data; only the canvas settings differ: herowrist (wrist big) and strip384 (three 512x384 cells in a row). MAST jobs: gh-mv-hero512-ptvc3r05, gh-mv-hero512-g6bnjmc5 (rl_research, 2026-09-28) and gh-mv-hero512-wlkdf20n (spatial_ai_research, 2026-10-04 to 10-11), using cosmos-framework-dev.

Training

• Initialization: weights only, with a fresh optimizer and scheduler and the step count reset to 0. The fork changed only the data and the canvas.
• Trained modules: the generation tower, timestep embedder, VAE/action projections and action embedding. The understanding and vision towers are frozen.
• Data: four sets, each episode with two third-person cameras and a wrist camera. Counterfactual and perturbed rollouts are included. Sampling is uniform over 296,942 episodes (digests in training/manifest.json):

• Sample: [calibration x 12] + [history] + [current + 16 future], each its own VAE item and action block. Calibration is used with p 0.9 (else black placeholders). History (stride 3, end-padded with the last real frame) is used with p 0.75, or forced with at least 9 real frames when calibration is absent.
• Actions: frame-wise end-effector deltas (Euler xyz, cm and degrees, plus gripper), not normalized. A null action replaces the future block with p 0.1 for action CFG.
• Augmentation and caption: axis relabelling of the actions (p 0.9) and photometric jitter. The caption is task-neutral, with no instructions.
• Optimization: AdamW at a constant LR of 5e-5, bf16, EMA off. 4 samples per GPU on 4 x 8 H100 (128 per iteration). Flow-matching loss at shift 5.
• Why step 52,000: it is the best evaluated checkpoint (42k / 52k / 68k) on RoboLab mv25. 68k only matched it, so the run was stopped at step 79,560 on 2026-10-11. The launch record is in training/ and the config in config.yaml.

Multi-view tiling

• Canvas: 4:3 cells of 256x192 px (hero layouts fix a 512-px short side). The tier is mv_hero_256_4x3, rendered at that exact size, giving 16x18 = 288 tokens per latent frame. Cameras are placed by role: tp0 is the main third-person camera, tp1 its <main>_1 sibling.
• Crops: every camera is cropped to 4:3, then resized. 16:9 sources (DROID 910x512, RoboLab official views) lose 114 columns per side. 512x512 sources lose 128 rows: centred for third-person views, bottom-anchored for the wrist (multiview_wrist_crop_y=1.0, which keeps the gripper).
• Training (one random plan per sample):
  • The wrist stays in slot 2 and the third-person cameras are shuffled over slots 0-1.
  • One camera is blacked out with p 0.10, two with p 0.05, never all.
  • Crop anchors are redrawn per camera: U[0.3, 0.65] for third-person, U[0.5, 1.0] for the wrist.
  • Photometric jitter is applied per tile, so empty slots stay black.
• Shared plan: the plan is drawn once per sample (GripperheadFDMDataset._sample_multiview_plan). The 12 calibration segments, the history and the current+future item all reuse it, so a camera never changes slot within a sample.
• Evaluation: the same code at is_train=False, with the fixed plan in the table, no dropout and anchors 0.5 / 1.0. An episode missing a camera is skipped.
• Caption: "A robot arm interacts with the scene. This video contains concatenated views from multiple camera perspectives. The video is 1.0 seconds long and is of 15 FPS. This video is of 576x512 resolution."
• Latent cap and shift: MAX_VAE_LATENT_SIDE=32 only sizes table-based position embeddings, so it is inert under Cosmos3-Nano's unified_3d_mrope. The 512-px short side selects shift bucket "480" (shift 5).

The figure shows real eval canvases (examples/eval_gripperhead_fdm_rollout.py::_read_eval_episode) from RoboLab mv25 episode_0000 (official cameras, BananaInBowlTask, frame 80) and episode_0019 (randomized cameras, PickOrangeObjectTask, frame 70). Each slot equals _view_tile of the raw frame byte for byte. The full spec is tiling/tiling.json, generated from the code.

Evaluation on RoboLab mv25

• Episodes: 25 RoboLab episodes not seen in training: 15 with the official 910x512 cameras, 10 with randomized 512x512 cameras.
• Protocol: 421 segments, each 2 rounds of 16 frames. The second round sees the model's own frames.
• Settings: action CFG 5, 35 UniPC steps, shift 5, with training-matched history, caption and CFG null.
• Metrics: segment-weighted. Per-role metrics use each camera's 256x192 4:3 field of view. The set is not part of this repo.

52k beats 42k on canvas PSNR in 20 of 25 episodes. 68k vs 52k is -0.05 dB (better in 11 of 25), a plateau. These numbers come from one seed and 25 episodes. Canvas metrics are dominated by the big cell, so compare layouts on the per-role columns.

Training losses

Mean of train@2_detail/flow_matching_loss_vision per window. The action loss is 0 because actions are only conditioning. These are training losses, not an evaluation.

Full curves are in metrics.jsonl, one record per 20 iterations up to 52,000.

Files

The frozen vision tower is not stored (FDM generation does not use it). No optimizer state is included.

Loading (cosmos-framework-dev)

hf download yyuncong/SyncWorld-multiview-hero512-step52000 --local-dir ./SyncWorld-multiview-hero512-step52000
# base config (docs/training.md, Step 2): python -m cosmos_framework.scripts.convert_model_to_dcp -o examples/checkpoints/Cosmos3-Nano --checkpoint-path Cosmos3-Nano

export WAN_VAE_PATH=examples/checkpoints/wan22_vae/Wan2.2_VAE.pth  # Wan2.2 VAE weights
export EVAL_QWEN_ASSETS=/path/to/qwen3vl_assets                    # offline only: Cosmos3-Nano processor files
export TORCH_HOME=/path/to/torch_home                              # LPIPS AlexNet weights (or pass --no-lpips)
PYTHONPATH=. python examples/eval_gripperhead_fdm_rollout.py \
  --checkpoint ./SyncWorld-multiview-hero512-step52000 --base-config-dir examples/checkpoints/Cosmos3-Nano \
  --eval-set ./robolab_mv25 --out ./eval_out --tag mv25-hero-i52k \
  --num-history-frames 25 --history-frame-stride 3 --num-pred-frames 16 --num-rollout-rounds 2 \
  --use-calibration --calib-segments 12 --calib-seg-len 5 --calib-frame-interval 3 \
  --action-convention backward_framewise --action-rot-format euler_xyz \
  --action-trans-scale 100 --action-rot-scale 57.2958 --action-norm none \
  --action-cfg-scale 5.0 --num-steps 35 --fps 15 --seed 0 \
  --multiview-layout hero --multiview-cell-aspect 4x3 --multiview-expect-tier mv_hero_256_4x3

• Base config: the eval builds Cosmos3-Nano from --base-config-dir (only model/config.json is read) and partially loads model/.
• Multiview flags: these must match training; --multiview-expect-tier fails fast otherwise.