SyncWorld multi-view (strip, 1536x384), step 31500

A SyncWorld forward-dynamics (FDM) world model on Cosmos3-Nano that sees three cameras at once. Two third-person cameras and the wrist camera are tiled side by side into one 1536x384 frame, and the model predicts the next 16 frames of that canvas. It is conditioned on the actions, a visual calibration clip (12 segments x 5 frames, interval 3) and a sparse history (up to 25 real frames, stride 3). Each camera keeps a full 512x384 cell, hence "strip384"; the hero layouts give two of the cameras only 256x192. Weights-only fork of the single-camera 512x512 run gh-cm-ax09-calib12-ph075-p7sc0z72 at iteration 174000. Siblings: hero512, herowrist512. Earlier checkpoint of this run: strip384-step22000.

MAST run gripperhead-mv-strip384-4x3-fork174k-4x8 (cosmos-framework-dev), trained in three stages:

• gh-mv-strip384-kh0nn226 / -mqv95pqw on rl_research (2026-09-28 to 09-30)
• -jzg9x1ch on spatial_ai_research (2026-10-07, resumed at 4000)
• -n2bvd345 on large_model_ablate (from 2026-10-08, still training at upload)

Training

• Initialization: all weights come from the source run, action layers included. Optimizer, scheduler and trainer state are reset, so the counter restarts at 0.
• Data: sampled uniformly over 296,942 episodes, each with two third-person cameras and a wrist camera (training/manifest.json). RoboLab is not used.
  • robocasa v2: 59,391 (12,237 expert + 47,154 counterfactual)
  • robomimic v2: 109,739 (16,608 expert + 93,131 counterfactual)
  • RLBench wrist-multiview: 58,104 (19,860 expert + 38,244 perturbed)
  • DROID 2-camera wide: 69,708 expert episodes (910x512)
• Recipe:
  • Forward dynamics; actions are backward_framewise euler_xyz deltas in cm/deg plus the gripper, unnormalized.
  • Calibration is included with p 0.9 and history with p 0.75; the future action is nulled with p 0.1 for CFG.
  • Augmentation: pose-axis augmentation p 0.9 (mode all), photometric jitter.
  • Optimizer: AdamW, constant LR 5e-5 (action layers x5), weight decay 0.05; bf16, EMA off.
  • Trained keys: moe_gen, time_embedder, vae2llm, llm2vae, action2llm, llm2action, action_modality_embed.
  • Compute: 4 nodes x 8 H100, 128 samples per iteration.
• Iteration 31500 was the latest checkpoint at upload; training continues. On RoboLab mv25 the run improves steadily up to 25000; from 25000 to 31500 the canvas metrics are flat and the per-camera PSNRs move by a few tenths of a dB between checkpoints. A later checkpoint (41500) is worse on the third-person views and SSIM, so the run has converged and 31500 is its best evaluated checkpoint, with the best mean per-camera PSNR of the multi-view layouts (see Evaluation). The launch record is in training/, the resolved config in config.yaml.

Multi-view tiling

tp0 is the main third-person camera (agentview_rgb, ...), tp1 its <main>_1 sibling. One frame is 48 x 12 latent tokens of 32 px, so slot seams fall on token boundaries.

• Crops: each camera is cut to 4:3, then resized.
  • 16:9 sources (DROID and the RoboLab official cameras, 910x512) keep the central 682x512 and are resized to 512x384.
  • 512x512 sources keep 384 rows, chosen by crop_y (0 = top, 1 = bottom).
  • At evaluation, crop_y is 0.5 for third-person cameras (rows 64-447) and 1.0 for the wrist (rows 128-511, keeps the gripper).
• Training plan (drawn once per sample):
  • The third-person cameras are shuffled over slots 0-1; the wrist stays in slot 2.
  • One camera is blacked out with p 0.10, two with p 0.05, so each camera is black in 6.7% of samples.
  • Each camera draws its own crop_y: U[0.3, 0.65] for third-person, U[0.5, 1.0] for the wrist.
  • Photometric jitter is one draw applied per tile, so black slots stay black.
• One plan per sample: the plan is drawn before any item is built. Calibration, history and current+future all reuse it, so a camera never moves within a sample. Calibration is tiled from the calibration directory's own cameras; if none of the surviving cameras are there, it becomes the null calibration.
• Evaluation plan: canonical tp0 | tp1 | wrist, nothing dropped, fixed crop_y, no jitter, also used for the calibration clip. An episode missing a camera is skipped.
• Caption (concat_view, same in training and evaluation): "A robot arm interacts with the scene. This video contains concatenated views from multiple camera perspectives. The video is 1.0 seconds long and is of 15 FPS. This video is of 384x1536 resolution."
• Latent cap, shift: MAX_VAE_LATENT_SIDE=64 (default 20), since the canvas is 48 tokens wide. Under unified 3D mRoPE the cap builds no position table, so evaluation needs no override. The 384 px short side falls in shift bucket "480" (shift 5).

The canvases above were built by the evaluation code path (_read_eval_episode) from two RoboLab mv25 episodes: episode_0000 (official cameras, BananaInBowlTask, frame 50) and episode_0019 (randomized cameras, PickOrangeObjectTask, frame 43). Dashed boxes are the per-camera metric regions. tiling/tiling.json holds every number in this section and was generated by importing the actual tiling, dataset and evaluation code (file hashes recorded).

Evaluation on RoboLab mv25

RoboLab mv25 has 25 held-out episodes recorded with Cosmos3-Nano-Policy-DROID: 15 with the official 16:9 cameras and 10 with randomized 512x512 cameras. Every 16 frames, two rounds of 16 frames are generated; the second round is closed-loop. Conditioning matches training: calibration 12x5@3, history 25@3, the training caption, and action CFG 5 with the future action nulled. Sampling uses 35 UniPC steps at shift 5. Scores are means over 421 segments. Per-camera PSNR is computed on a common 256x192 4:3 view and is comparable across layouts; canvas metrics are not.

Paired per episode (mean per-episode change, episodes improved out of 25):

• 22000 to 25000: every metric improves: canvas PSNR +0.29 dB (22/25), SSIM +0.009 (25/25), LPIPS -0.004 (23/25), tp0 +0.36 dB (22/25), tp1 +0.29 dB (21/25), wrist +0.19 dB (20/25).
• 25000 to 28500: mixed. Canvas PSNR +0.10 dB (16/25) and wrist +0.36 dB (21/25), but the third-person views dip (tp0 -0.16 dB, 8/25; tp1 -0.23 dB, 9/25) and SSIM is -0.002 (6/25); the mean per-camera PSNR is flat (-0.01 dB).
• 28500 to 31500: tp0 +0.42 dB (22/25) and tp1 +0.19 dB (16/25), wrist -0.25 dB (5/25); canvas PSNR -0.02 dB (11/25). Overall, 25000 to 31500 changes canvas PSNR by +0.08 dB (16/25) and the mean per-camera PSNR by +0.11 dB (16/25): a plateau with small checkpoint-to-checkpoint oscillations between the cameras (constant LR, no EMA).
• 31500 to 41500: the third-person views drop (tp0 -0.63 dB, 3/25; tp1 -0.65 dB, 4/25) and SSIM falls by 0.013 (1/25) while the wrist gains +0.31 dB (22/25); the mean per-camera PSNR is -0.32 dB (5/25). Past 31500 the run trades third-person quality for the wrist.
• 31500 vs strip256 at 64000: tp0 +3.0 dB and tp1 +3.4 dB (25/25 each), wrist +1.2 dB (25/25). The larger cells explain part of this.
• vs the siblings (at 31500): mean per-camera PSNR is 27.9, against 26.8 for hero at 52000 and 26.0 for hero_wrist at 56000. tp1 is best here (32.0, against 29.0 and 28.2); tp0 now matches hero (30.4), and hero_wrist still leads on the wrist (22.2 vs 21.5), the camera it puts in its big cell.

The official/randomized gap mixes camera geometry with source aspect. 8 of the 25 recordings are policy successes.

Training losses

Mean train@2_detail/flow_matching_loss_vision over the 1000 iterations before each mark, on the training mix. FDM has no action loss, so this is also train@2/loss. The loss at iteration 20 is 0.0680. metrics.jsonl logs every 20 iterations, and a resumed attempt's records replace those of the preempted attempt.

Files

Loading (cosmos-framework-dev)

hf download yyuncong/SyncWorld-multiview-strip384-step31500 --local-dir ./SyncWorld-multiview-strip384-step31500
# base config (docs/training.md, Step 2): python -m cosmos_framework.scripts.convert_model_to_dcp -o examples/checkpoints/Cosmos3-Nano --checkpoint-path Cosmos3-Nano
export WAN_VAE_PATH=/path/to/Wan2.2_VAE.pth      # local Wan2.2 VAE
export TORCH_HOME=/path/to/torch_home            # LPIPS AlexNet in $TORCH_HOME/hub/checkpoints (or --no-lpips)
export EVAL_QWEN_ASSETS=/path/to/qwen3vl_assets  # offline only: local Cosmos3-Nano processor files
PYTHONPATH=. python examples/eval_gripperhead_fdm_rollout.py \
  --checkpoint ./SyncWorld-multiview-strip384-step31500 --base-config-dir examples/checkpoints/Cosmos3-Nano \
  --eval-set ./robolab_mv25 --out outputs/mv25 --tag strip384-step31500 \
  --num-history-frames 25 --history-frame-stride 3 --num-pred-frames 16 \
  --use-calibration --calib-segments 12 --calib-seg-len 5 --calib-frame-interval 3 \
  --action-convention backward_framewise --action-rot-format euler_xyz \
  --action-trans-scale 100 --action-rot-scale 57.2958 --action-norm none \
  --action-cfg-scale 5.0 --num-steps 35 --fps 15 \
  --multiview-layout strip --multiview-cell-size 512 --multiview-cell-aspect 4x3 \
  --multiview-expect-tier mv_strip3_512_4x3

The repo has no config.json, so the eval builds Cosmos3-Nano from --base-config-dir and loads model/ partially. FDM does not use the vision tower. The remaining defaults reproduce the table: 2 rounds, shift 5, and the training history padding, caption and CFG null. Leave GRIPPERHEAD_EVAL_USE_EMA unset. --multiview-expect-tier rejects any other canvas. Results land in <out>/<tag>/mv_strip3_512_4x3/. RoboLab mv25 is internal: samworlds_mast/tree/users/yuncongyang/eval_sets/robolab_mv25.