SyncWorld exp-b7, in-context retrieval comparison, method C (library), update 8000

One of the four runs of SyncWorld-dev's retrieval comparison (in-context branch, wm_ttt/retrieval/README.md, upstream commit 827ab2a). Full-attention in_context layout, no TTT memory: a query is one anchor frame plus its own 16 future commanded actions, and the model generates the next 16 frames. Before the query, K observed 16-transition video/action chunks (K uniform over 1..8, one K per update for all GPUs) sit in the same attention window as clean conditioning items. Every stream applies one random action-axis relabel to its recording, so the anchor-only query can learn the relabelled convention only from the chunks. The four runs differ only in which K - 1 chunks lead the query:

This repo: C, library. Its chunks: the adjacent chunk plus the nearest chunks among the query's own earlier grid chunks and a library of every chunk of every other recording of the mixture (all datasets and cameras), by commanded-action curve, start pose in each chunk's own camera frame, DINOv2-base image similarity to the anchor and start gripper state (each over the library's median scale); at most 2 chunks per other recording. Recipe wm_ttt/retrieval/recipe_qctx_incontext_C_library_8gpu.toml (training/recipe.toml); run qctx_incontext_C_library_8gpu_main.

Training

• Shared by the four runs: exp-b7 initialization (MAST run gripperhead-nocalib-droidmix-ax06-hist25-512-4x8, iter 76000); trains moe_gen, the time embedder and the VAE/action projections (B/C/D also their context-gap embedding); AdamW, LR 5e-5 after a 400-update linear warm-up, then constant; weight decay 0.05; the backbone's default noise schedule and an unweighted video loss; 8 GPUs x 1 stream = 8 streams per update; 8000 updates; seed 42, so all four runs drew the same queries, K values and noise levels.
• Data: formal_2 (the reference two-camera index, 10,478 setups), robomimic v2 and DROID pairs (droid_fdm_2cam_wide_pairs_shards, train split, recorded at 15 Hz), both third-person cameras of every dataset, mixed 1:1:1: 110,562 recordings with eight full chunks before an anchor. The training mixture (SHA-256 a1e74e73..., training/mixture.json) was derived on every node from our qctx_v1 bundle: camera 2 on the recording's own directory (as upstream's prepare_formal2_views.py), DROID's manifest fps 20 with recorded_fps 15 (frames are never resampled; conditioning is 15 fps), DROID camera poses from its extrinsics.
• Library (training/library_summary.json): 2,588,638 16-frame chunks of 142,012 recordings (formal_2 20,956, robomimic 31,032, DROID 90,024; both cameras), DINOv2-base CLS features, median scales 49.83 cm / 89.96 deg / image 0.709, library.json SHA-256 e0849267514d0f66..., built from this run's mixture.
• Compute: 1 node x 8 B200 on MAST (spatial_ai_research), about 1.7 s per update, no restarts, finished 2026-10-22 09:22 PDT.

Training loss

train@2/loss (the query's video flow-matching loss) averaged over the 500 updates before each mark:

Note: the loss is lowest during the warm-up, while the model is still essentially exp-b7, and settles about 15-20% higher once the LR reaches 5e-5 (around update 1000), after which it is flat. Controlled for noise level and K this holds in all four runs, for every K, and it is not a sampling or logging artifact. Every query is drawn fresh (no epochs), so this is effectively a held-out loss on the training distribution. The cause is most likely the LR of 5e-5 with only 8 streams per update. Compare these checkpoints with exp-b7 on held-out data before relying on them. D's random library chunks cost about 4% (more at larger K) relative to A from the start.

Files

The trainer kept FP32 master weights (60.7 GB); this copy is cast to bf16, the model's compute precision. The FP32 weights and the full training state (optimizer, scheduler, data loader) of updates 2000, 4000, 6000, 7000 and 8000 are on Manifold: samworlds_mast/tree/users/yuncongyang/syncworld_ttt/outputs/qctx_incontext_C_library_8gpu_main/checkpoints/.

Loading (SyncWorld-dev, in-context branch)

hf download yyuncong/SyncWorld-b7-incontext-C-library-step8000 --local-dir ./SyncWorld-b7-incontext-C-library-step8000
python examples/eval_gripperhead_fdm_rollout.py --checkpoint ./SyncWorld-b7-incontext-C-library-step8000 ...   # repo dir = iter dir

As a warm start, point checkpoint.load_path (or train_query_context.sh --init-checkpoint) at the repo directory.