student_prefix_polaris_15K_olmo3-7b_continual_Q_qwen3-30b-a3b_cutoff4096_epoch_1_mask
Online ROSE across a vocabulary boundary. Olmo-3-7B-Think-SFT (student, vocab 100278) trained on continuations written by Qwen3-30B-A3B-Thinking-2507 (teacher, vocab 151936), with cross-entropy applied only to the teacher's tokens.
prompt ---- student prefix (4096 tok) ------- teacher continuation (~941 tok) ----
response_mask 0 response_mask 1
context, on-policy the target, off-policy
The two vocabularies are different, so the boundary is crossed as text: the student's
prefix is decoded, the teacher is shown it under its own chat template, and the reply is
tokenised back into the student's vocabulary. The mask boundary is len(prefix_ids) by
construction — the student's ids are never re-encoded.
Configuration
| student | allenai/Olmo-3-7B-Think-SFT — 7.30B, vocab 100278, no GQA, 24 sliding / 8 full attention |
| teacher | Qwen/Qwen3-30B-A3B-Thinking-2507 — MoE, vocab 151936, 2 vLLM TP=1 replicas |
| data | SeanWang0027/polaris_hard, 15,368 competition-maths problems, 1 epoch |
| cut | K = 4096 prefix + t = 1024 teacher tokens → max_response_length 5120 |
| schedule | asynchronous, depth 3 |
| batch | train_batch_size 64, n 4 → 256 rows per block |
| optimizer | AdamW, lr 1e-5 constant, no warmup, wd 0.01, clip 1.0, bf16 |
| steps | 240 (one epoch), global_step_240 |
| hardware | 10 × NVIDIA GH200 on TACC Vista — 2 teacher + 8 FSDP trainer ranks |
| wall clock | 8 h 37 m, 129 s/step |
Code: cl-from-nothing/online-rose @ 4264fe9.
Every rollout of this run is published as
SeanWang0027/polaris_rose_rollouts_olmo3-7b_from_qwen3-30b-a3b_cutoff4096_240steps
— 61,440 rows with the teacher's actual continuation and the full token accounting.
What actually happened during training
The objective moved.
actor/ce_nll_before, mean of first 20 steps |
0.6398 |
actor/ce_nll_before, mean of last 20 steps |
0.5425 (−15.2%) |
actor/grad_norm |
1.10 → 0.20 |
rose/pool_failures |
0 across all 240 steps (58,624 teacher calls) |
rose/teacher_hidden_frac |
1.000 — the teacher was never the bottleneck |
rose/seam_shift |
0 on all 61,440 rows |
rose/teacher_finished |
2.07% |
rose/empty |
4.17% |
But 84.5% of the weights never moved a single bit. Measured against the base model:
| this run | a same-family control run | |
|---|---|---|
| parameters that changed at all | 15.46% (1.13 B / 7.30 B) | 12.32% |
| global relative L2 drift | 2.33e-3 | 1.75e-3 |
| largest per-tensor drift | 8.57e-3 (layers.31.self_attn.q_proj) |
3.36e-3 |
ce_nll_before over the run |
−15.2% | +0.3% (flat) |
The control is a Qwen3-4B ← Qwen3-32B same-vocabulary run on the same corpus, K, t and schedule (checkpoint); the student and teacher differ, so it bounds nothing on its own — it is context, not a matched comparison.
This run moved measurably more than that one, in the direction the loss moved. The ceiling is
the same, though, and it is worth stating: the launcher passes
actor_rollout_ref.actor.fsdp_config.model_dtype=bfloat16, which overrides the torch.float32
verl otherwise forces for the actor — and verl's own comment at fsdp_workers.py:302 reads
"we have to create model in fp32. Otherwise, the optimizer is in bf16, which is incorrect."
With bf16 master weights and bf16 Adam moments, one step at lr = 1e-5 is the same order as a
bf16 half-ULP at |w| ≈ 0.02, so most weights cannot move. The repo keeps bf16 deliberately, so
that every number in it stays comparable; an fp32 arm is a separate control that has not been
run.
No downstream evaluation has been run. Neither this checkpoint nor
allenai/Olmo-3-7B-Think-SFT has been measured on AIME 2025 or anything else. The reward
logged during training is log-only — compute_policy_loss_ce ignores advantages — and it
scores a truncated prefix spliced to a partial continuation.
Treat this as a reproduction artifact of the training run. A falling CE on the teacher's tokens is not a claim about downstream ability. If you want that comparison, run both models under identical settings.
Two numbers that are properties of the configuration, not defects
rose/teacher_tokensaverages 940.6, not the configured 1024. The teacher generates 1024 tokens in its vocabulary; re-encoded into the student's they are fewer. Read the supervised length from the metric, never fromt.rose/teacher_finishedis 2.07%. After a 4096-token prefix, 1024 more tokens almost never reach the end of a competition-maths solution, so the student is overwhelmingly taught to continue and only rarely to stop.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "SeanWang0027/student_prefix_polaris_15K_olmo3-7b_continual_Q_qwen3-30b-a3b_cutoff4096_epoch_1_mask"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype="bfloat16", device_map="auto")