SyncWorld exp-b7 + query-context TTT memory, two stages (4000 + 4000 updates), robocasa + robomimic + DROID

The wm_ttt query-context line (SyncWorld-dev wm_ttt/README.md, section 8; wm_ttt/train/run_query_context.sh) trained from exp-b7 on three datasets. The model gets no history: a query is one anchor frame plus its own 16 future actions. Everything before the anchor enters through the test-time-training (TTT) memory. Each training stream takes one recording under one camera, draws one random action-axis relabel for it, and walks the eight contiguous 16-transition chunks that end at the anchor: the first chunk is only written, every later chunk is first predicted from the memory of the chunks before it and then written, and the anchor-only query is predicted last (K = 1..8).

MAST run qctx_large (SyncWorld-dev, 2026-10-17): stage 1 run qctx_nextchunk_expb7_stage1_tttactproj_8gpu_qctx_large_s1, stage 2 run qctx_nextchunk_stage2_joint_bb1e5_const_8gpu_qctx_large_s2. This repo holds stage 2's final update (4000), i.e. 8000 updates in total.

Training

• Initialization: stage 1 from exp-b7 (fresh TTT branch, optimizer and data stream). Stage 2 loads only stage 1's final weights (TTT, backbone, interfaces); its optimizer, schedule and data stream start fresh.
• Data: a query_context_mixture_v1 of three components, sampled 1:1:1 (a stream draws a component by weight, then a setup, a recording and an anchor uniformly). Both third-person cameras of every dataset; a stream decodes one camera throughout. Recordings count only if they have an anchor after eight full 16-transition chunks.
  • robocasa formal_2: the reference two-camera index (prepare_formal2_views.py, 10478 setups, content digest equal to the reference; evaluation-cohort task scenes excluded): 18674 recordings.
  • robomimic v2, both cameras: 15516 setups, 28894 recordings.
  • DROID pairs (droid_fdm_2cam_wide_pairs_shards, train split), both cameras: 45012 setups, 62994 recordings. Recorded at 15 fps; frames are never resampled (the model is conditioned at 15 fps).
• Compute: 1 node x 8 B200 on MAST (spatial_ai_research), 8 streams per update (no accumulation). Stage 1 5.2 s per update, 43 GiB peak per GPU; stage 2 5.5 s per update, 65 GiB. No preemption; both stages in 12.5 h.
• W&B was off (the recipes' default). The exact recipes, launch records and the data mixture digests are in training/stage1 and training/stage2; the resolved stage-2 config is config.yaml.

Training loss

Mean of train@2/loss (the motion-weighted vision flow-matching loss of the K = 1..8 next-chunk reads and the query) over the 400 updates before each mark. Training loss on the three-dataset mix, not a held-out evaluation (the line's evaluation is the K8 score in dB on held-out formal_2 recordings, not run here).

Full curves: metrics.jsonl (stage 2) and training/stage1/metrics.jsonl.

Files

The trainer's model checkpoint keeps FP32 master weights for the trained parameters (46.8 GB). This copy is cast to bf16, the model's compute precision. The FP32 weights and the full training state (optimizer, scheduler, data loader) of both stages are on Manifold, samworlds_mast/tree/users/yuncongyang/syncworld_ttt/outputs/: qctx_nextchunk_stage2_joint_bb1e5_const_8gpu_qctx_large_s2/checkpoints/iter_000004000/ (this model) and qctx_nextchunk_expb7_stage1_tttactproj_8gpu_qctx_large_s1/checkpoints/iter_000004000/ (stage 1's final), plus every 800th update of each stage.

Loading (SyncWorld-dev)

hf download yyuncong/SyncWorld-b7-qctx-nextchunk-step8000 --local-dir ./SyncWorld-b7-qctx-nextchunk-step8000
python examples/eval_gripperhead_fdm_rollout.py --checkpoint ./SyncWorld-b7-qctx-nextchunk-step8000 ...   # repo dir = iter dir

As a warm start, point checkpoint.load_path (or a launcher's --init-checkpoint) at the repo directory. train_query_context.sh --stage 2 additionally checks the standard run layout: place the files as <run>/checkpoints/iter_000004000/ with config.yaml also in <run>/.