Qwen3.8-27B Longbook27 gearbox slots (retrained Gated DeltaNet mixers)

What the files are

train_v2/slots.safetensors, train_v3/slots.safetensors and blend_a0.5/slots.safetensors (current best, see below) each hold 16 Gated DeltaNet token mixers, one for each full-attention ("slot") layer of Qwen3.8-27B: layers 3, 7, 11, ..., 63. They are not a standalone model. A runtime gearbox keeps both mixers loaded in every slot (A = Qwen's own attention, D = this DeltaNet) and picks one per slot at runtime; gear k = k slots running D. Gear 0 is bit-identical to stock Qwen3.8-27B.

Tensor names follow the stock Qwen3_5GatedDeltaNet module, e.g. for layer 3: layers.3.linear_attn.{A_log, dt_bias, conv1d.weight, in_proj_qkv.weight, in_proj_z.weight, in_proj_a.weight, in_proj_b.weight, norm.weight, out_proj.weight} (9 tensors x 16 layers). Safetensors metadata records the step, recipe, base and init.

Not included: the Longbook LoRA. Longbook's stage-B rank-32 LoRA is summerAIO's work, not ours; get it from their repo. Gear 16 + that LoRA approximates Longbook as released.

Attribution

• summerAIO/Qwen3.8-27B-Longbook27-v2 @ 46319430c16af66860b20f14986ad1032e991796, Apache-2.0. Source of the slot architecture (Gated DeltaNet in the 16 full-attention layers) and of the original mixers, which were the initialisation for v2.
• Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, Apache-2.0. Base model; all non-slot weights and the attention targets the mixers were distilled toward.

These files are modified derivatives of Longbook's mixers and are released under Apache-2.0.

Training recipe (stage A')

Each D mixer is distilled toward the unsteered base model's attention output on the same normed input, loss nmse + 0.1·cos (Longbook's stage-A loss), 8k-token sequences.

• v2: 1000 steps, initialised from Longbook's own mixers.
• v3: +2000 steps from v2: fresh cosine schedule at half v2's peak lr (lr 1e-4, warmup 30), seed 1102, constant 8k.

Data (26M tokens, public): UltraChat-200k (MIT), OASST2 (Apache-2.0), OpenR1-Math-220k (Apache-2.0), NVIDIA OpenCodeReasoning (CC-BY-4.0), Wikipedia 20231101 en/ja (CC-BY-SA-3.0). Evaluation articles were excluded from the Wikipedia pool.

Measured results

The following is copied verbatim from our share note (LONGBOOK-SHARE.md, which also renders the image above).

1. How we use it

• We keep both mixers loaded in each of the 16 slots and pick one at runtime: A = Qwen attention, D = Longbook DeltaNet. Gear k means k slots run D.
• Gear 0 is bit-identical to stock Qwen3.8-27B (CPU smoke test, test_gearbox.py).
• Down-shift (A to D) is instant, because D runs on every token and its recurrent state stays warm. Up-shift (D to A) is either soft (A kept writing K/V) or hard (drop the slot's KV and re-prefill).
• Gear 16 + LoRA is approximately Longbook as released.
• Literature check (LIT-REVIEW.md): conversion and trained per-token routing are prior art. We found no prior work on runtime switching between two independently trained mixers with a warm state (moderate confidence).

2. What we measured (E-LB1 gear ladder)

These are fixed-gear prefill measurements. KL is averaged over 3 chat conversations (32k tokens). Needles are 18 exact passkeys at 4k, 16k and 64k.

• Low gears hold. k1 and k2 keep 18/18 needles up to 64k. k4 is the edge, and k8 loses every needle.
• All-D fails on our chat text. All 16 slots in D plus the LoRA (our reconstruction of Longbook) gives KL 1.61 and 0/18 needles, and it already fails at 4k. Longbook's own report gives KL 0.207, measured on Wikipedia.
• Caveat. Our base is NVFP4/FP8-dequantised. The LoRA is a rank-32 truncated-SVD reconstruction of their stage-B change, capturing 98.4-99.9% of its energy per tensor. So this is not a bit-exact replay of their bf16 checkpoint. We trained no LoRA.

3. What we changed

Retrained D slots (stage A'). We retrained Longbook's 16 D mixers toward the unsteered base model's attention output, using nmse + 0.1·cos (Longbook's stage-A loss). The data is 26M tokens of public chat, math, code and Wikipedia: UltraChat, OASST2, OpenR1-Math, OpenCodeReasoning and Wikipedia en/ja.

• Per-slot cosine to attention on held-out chat that was never trained on: 0.42 (Longbook) -> 0.79 (v2, 1000 steps) -> 0.82 (v3, 2000 more steps from v2 at 8k).
• The v3 value is from the last validation row (step 2000). At writing, v3 had no DONE marker yet (export check pending).

v2 ship ladder (partial). Gear 4 here uses the cheapest single slots (L55, L59, L39, L51); gear 6 has not been run.

• KL to gear 0 fell 32% on chat, 24% on code and 39% on Wikipedia.
• Needles: 4k 5/6 -> 2/6, 16k 3/6 -> 3/6, 64k 1/6 -> 2/6.
• This misses the 40% bar, so v2 does not ship. The v3 ladder is in progress.

Grounded-QA eval set (grounded_qa/, data/qa_manifest.json):

• dev has 216 items and release has 594 (72 / 198 questions). Each question also has a closed-book control and a wrong-context control.
• Each item puts one gold document at depth 0.1 / 0.5 / 0.9 of an 8k / 16k / 64k prompt.
• Gold documents are post-cutoff (after 2026-08-05): new Wikipedia articles, CC-BY arXiv papers and RFCs. They pass a 13-gram contamination check against our training pools.
• The splits are disjoint: 0 shared documents and 0 shared questions.

Ship rule (pre-registered). A stage ships only if it beats the previous stage at gears 4 and 6 on chat, code and Wikipedia:

• KL to gear 0 drops by at least 40%;
• needles do no worse;
• no data source regresses.

Status update (2026-11-02, after the note above was written). The v3 export finished (DONE marker 2026-11-02T06:43Z, export verified) and the v3 ship ladder has been measured: SHIP: NO. See Known limitations.

Current best (2026-11-02): blend a=0.5 of Longbook's slots and v3 (best so far, does not pass the ship rule)

Best so far, does not pass the ship rule (pre-registered 40% KL bar: gear 4 chat -29%, code -22%, wiki -34%). Same architecture as Longbook; our retrained slots blended with theirs; better fidelity to the base model at every gear we measured (4 and 6); needle retrieval preserved to 64k.

blend_a0.5/slots.safetensors = 0.5 x v1 + 0.5 x v3, elementwise, for every tensor (merge_slots.py). v1 is Longbook's own slots; v3 descends from v1, so both sit in the same basin. Source: monkey/longbook/README.md, section "Weight blend v1 <-> v3", and blend_a0.5/ship.txt. Measured-order gears, LoRA off; KL change is relative to v1.

Answer-span KL / top-1 agreement vs gear 0 on the 8k dev QA, gold document vs no document (closed):

Leave-one-out from the gear-4 set (source: monkey/longbook/README.md, section "Leave-one-out"). 8k dev QA exact / 24 and answer NLL vs gear 0 (gear 0: 20/24), with slots 55, 59, 39 and 51 in D except the one kept on attention:

The retrieval loss is a three-way interaction among the late slots 51, 55 and 59. Keeping any one of 51/55/59 on attention restores 18-21/24 vs stock 20/24, for v1 and v3 alike; L39 does not matter.

Layer-wise merge (tested, not included here): SHIP: NO. v1 weights on slots 51, 55 and 59, v3 everywhere else (source: results/merge-layer/ship.txt). KL drops vs v1 at gear 4: chat 4%, code 1%, wiki 16% (gear 6: 13%, 5%, 25%). Needles at gear 4 are 4/6, 5/6, 4/6, so 4k is below v1's 5/6. At gear 6 they are 5/6, 5/6, 3/6, below v1 at 16k and 64k. 8k QA is 14/24 at gear 4 and 11/24 at gear 6. It keeps QA but gives up almost all of v3's KL gain, so the uniform a=0.5 blend stays the better trade.

Caveats:

• Needles are 6 per length, so the needle gains are suggestive; QA is the steadier signal.
• 8k QA 13/24 vs v1 15/24; paired answer NLL +0.08 (95% CI +0.006 to +0.145), mildly adverse, n=24.
• Gear-6 needles for the blend, 5/6, 6/6, 6/6, meet the v1 baseline.
• Still SHIP: NO. The blend fails all six source/gear KL-drop checks (chat, code and Wikipedia at gears 4 and 6) against the 40% bar. Code is the largest shortfall (-22% / -23%). Even v3 reaches only -32% / -34% on code, and no tested blend passes.
• Fixed-gear prefill measurements only, on an NVFP4/FP8-dequantised base (see the caveat above).

Known limitations

None of the files here passes our pre-registered ship rule. The retrained sets v2 and v3 on their own lose long-context retrieval at gears 4 and 6. The a=0.5 blend above recovers most of it (needles and QA near v1) but gives up part of the KL gain. Treat all three as research artefacts. The numbers below compare v1 (Longbook's own D slots) with v3, using the measured gear order (slots 55, 59, 39, 51, 7, 43, ...) with the LoRA off.

• Ship rule fails. The rule needs KL to gear 0 to drop by at least 40% on chat, code and Wikipedia at gears 4 and 6, with needles no worse and no source regressing. v3 gets chat -37.5% / code -32.5% at gear 4 and chat -39.6% / code -34.0% at gear 6. Only Wikipedia passes (-45.5%, -47.8%). (v2 at gear 4: chat -32%, code -24%.)
• Retrieval regresses.
  • Needles: 4k exact passkeys fall from 5/6 to 1/6 at gears 4 and 6. At gear 6, 16k goes 6/6 -> 4/6 and 64k 6/6 -> 2/6.
  • 8k grounded QA (dev set, answer span teacher-forced, exact means every answer token is the argmax): v3 gear 4 gets 6/24 exact against 20/24 at gear 0. v1 gear 4 gets 15/24.
  • Answer-span KL to gear 0 at gear 4, 8k, with the gold document: 0.41 (v1) -> 0.81 (v3), paired difference +0.40 +-0.19.
• Average fidelity still improves. Held-out per-slot fit to attention (val nmse) goes v1 0.806 -> v2 0.349 -> v3 0.292. KL drops on every source. With no document in context, answer-span KL at gear 4, 8k, improves 0.22 -> 0.13.
• No single slot carries retrieval (single-slot ablation, 24 dev QA at 8k).
  • Any one slot alone in D leaves QA at gear-0 level (19-22/24). Only L59 shows a visible cost: dNLL +0.021 for v1 and +0.065 for v3.
  • The damage is super-additive: v3's single-slot costs for slots 55, 59, 39 and 51 sum to about +0.05 answer NLL, but the four together cost +0.84 (6/24). Retrieval is redundant across the attention layers, and converting several removes that redundancy.
• Diagnosis. The training objective is per-token nmse to the attention output, averaged over generic text. It rewards imitating attention on average, not retrieval: it rarely sees a token that needs an exact long-range lookup, so it cannot protect one. Consistent with this, v3 matches attention better than v1 when the answer is parametric (no document) and worse when the answer must be copied from the context. The state-allocation mechanism behind this is a hypothesis we have not tested. We have stopped training further with this recipe; blending with v1 (above) is how we currently trade the two off.

Full tables, confidence intervals and the follow-up plan are in monkey/longbook/README.md of the Atlas3DSS/neuropedia repo (github.com/Atlas3DSS/neuropedia, private at the time of writing). The relevant sections are "Stage A' 8k results: v1 / v2 / v3 (2026-11-02) — SHIP: NO" and "Single-slot retrieval ablation".

How to load

The loader is monkey/longbook/gearbox.py in our neuropedia repo (github.com/Atlas3DSS/neuropedia; private at the time of writing). Usage, from its docstring:

gb = Gearbox(oracle.lm, slots_path, lora_path)   # lm = HF Qwen3.8-27B; slots_path = one of these files
gb.set_gear(['L3', 'L11'])      # these slots use D; the rest use A        (gear 0 = set_gear([]))
gb.probe = True                 # also run the non-selected path at every slot and record the A/D gap
gb.set_lora(True)               # optional: Longbook's stage-B LoRA (theirs, not in this repo)

Without the gearbox: build a stock Qwen3_5GatedDeltaNet for each slot layer with Qwen3.8-27B's linear-attention config and load layers.<L>.linear_attn.* from the file into it. Limits of our implementation: D runs prefill only (no cache) and without a padding mask, so batch rows must be unpadded and of equal length.