SyncWorld three-camera strip384, simple_a3 views first (10 deg / 10 cm), update 17000

Status: intermediate checkpoint of a run still training (update 17000 of 30000).

One of the two runs of SyncWorld-dev's three-camera comparison (in-context branch, wm_ttt/retrieval/README.md "Three-camera comparison", upstream commit 40be701 plus local fixes): the simplified nearest retrieval with joint camera views, views first, at 10 deg / 10 cm (this repo, the main method) against recent8 (yyuncong/SyncWorld-mv3-strip384-recent8-step*). Run qctx_incontext_mv_simple_a3_views_first_8gpu_4n30k_s384_v10, recipe wm_ttt/retrieval/recipe_qctx_incontext_mv_simple_a3_views_first_8gpu.toml (training/recipe.toml).

• Frames: every frame is a 1536x384 strip of three 512x384 (4:3) cells: the recording's two third-person cameras, shuffled between the first two cells per stream, and the wrist camera in the last, with per-stream view dropout (one camera 10 %, two 5 %) and random vertical crops (third person 0.3-0.65, wrist 0.5-1.0), as cosmos-framework-dev's strip384 run. One tiling plan per stream holds for the query, its own chunks and every library chunk.
• Layout: full-attention in_context, no TTT memory. A query is one anchor frame plus its own 16 future commanded actions, and the model generates the next 16 frames; before it, K observed 16-transition video/action chunks (K uniform over 1..8, one K per update for all GPUs) sit in the same attention window as clean conditioning.
• Retrieval: the adjacent chunk, then K - 1 chunks chosen by the simplified nearest rule (commanded-action curve with step-wise gripper states and the end-effector trajectory relative to each chunk's start, action weight 3; start pose; DINOv2-base appearance) among the query's own earlier chunks and a library of every 16-frame chunk of every other recording, at most 2 per recording, never the query's own episode (all RLBench renders of a trajectory are one episode).
• Views first at 10 deg / 10 cm: a library chunk is scored on both shown third-person cameras jointly. It enters through the query camera its own camera lies within 10 deg of rotation and 10 cm of position of (same field of view, same dataset), and is admitted only when its recording also fills the query's other shown camera within the same limit, so no cell is ever black or shows a mismatched view; the query's own earlier chunks fill the remaining slots. At this limit robocasa / robomimic queries mostly retrieve their camera-rig twin episode.
• Library (training/library_summary.json): mv_strip_v2_v10_views_grip_anch, 3,852,002 chunks of 214,336 camera recordings (both third-person cameras of every leaf), camera poses relative to the robot base, the gripper and anchored-trajectory variants; built from the training prep, view limits recorded in the library and checked at training start.

Training

• Initialization: cosmos-framework-dev's strip384 run (gripperhead-mv-strip384-4x3-fork174k-4x8, iter 43500), already trained on these four datasets as 1536x384 strips.
• Optimization (the recipe as committed): AdamW (betas 0.9 / 0.99, weight decay 0.05), LR 5e-5 after a 400-update linear warm-up, then constant; trains moe_gen, the time embedder, the VAE/action projections and the context-gap embedding; one action-axis relabel per stream (p 1.0); bf16 compute with fp32 master weights for the optimizer; full activation checkpointing.
• Scale: 4 nodes x 8 H100 (HSDP: an 8-way FSDP group per node, 4 replicas), 1 stream per GPU = 32 streams per update; 30,000 updates; a checkpoint every 1000.
• Data (syncworld_ttt/data/mv_strip_v2, media streamed from the shard sets): robocasa_v2 10,478 recordings (the formal_2 reference units only), robomimic_v2 16,608, rlbench_wrist_mv 11,314 (1,061 distinct trajectories over 8 render sets), droid 68,768 (recordings over 3,000 frames dropped); expert recordings only; component weights robocasa 2 : robomimic 2 : rlbench 1 : droid 2.
• Compute: MAST (spatial_ai_research), conda cosmos_fdm_conda_rlr:98a0b25c.

Training loss

train@2/loss (the query's video flow-matching loss) averaged over the 500 updates before each mark:

Files

• model/: the trainer's DCP model state, cast to bf16 (every tensor checked after reload).
• config.pkl: the trainer's config.
• training/recipe.toml: the recipe; training/library_summary.json: the chunk library it retrieved from.