Whittle-Next-26B-A3B

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

Whittle-Next is a Qwen3.8-Flash-Next-format model (qwen4_exp) built from Qwen3.6 — something in between the two generations, hence the name. It runs on stock llama.cpp, no patches, no --override-kv. This is the precursor to the 27B. Same recipe, half the memory table, one hour of online distillation. This build stays published so the format, the serving path and the numbers can be checked now.

Status

Research preview, superseded. Successor (recommended): Whittle-Next-27B-A3B — same body, 2B-row memory trained in the memorisation regime, hyper-connection streams unlocked, LoRA on layers 2–12, reasoning distill on complete thinking traces. Read its caveats; for maths this 26B is marginally better. The 27B is out and is the one to run: Whittle-Next-27B-A3B (v3: 2B-row table trained in the memorisation regime, layer-2–12 LoRA, reasoning distill on complete thinking traces; GSM8K 86.5% / 86.0%, stop probe 12/12). The 27B is in turn the parent of the current flagship, Whittle-Qwen-3.8-35B-A3B.

What this is, and is not. These are not finished, general-purpose models. They are working starting points for anyone who wants to build on the Qwen4-Next (qwen4_exp) architecture — a body that already loads and runs on stock llama.cpp, with hyper-connections, a hashed n-gram memory and per-layer-embedding tensors wired in, plus every piece needed to keep training them (frozen body, trainable checkpoint, table, hash contract, trainer, exporter). Treat the numbers as a baseline to improve on, not a product.

Measured

How it was made

The lineage is not a secret and is part of the story: Qwen3.6-35B-A3B is the body, Qwen3.8-27B is the teacher, Qwen3.8-Flash-Next is the architecture target. All Apache-2.0.

1. Prune. Qwen3.6-35B-A3B, 256 → 180 routed experts per layer, scored one-shot with a gate-free activation-mass criterion (82% of routed mass kept). 34.7B → 25.1B.
2. Heal. Self-distillation from the unpruned weights through a routing mask (same weights, mask on/off), LoRA on the kept experts, Muon + AdamW. Result published as Qwen3.6-Whittle-25B-A3B (GSM8K 92.5% in transformers).
3. Sigmoid gate conversion. The Flash-Next format uses a sigmoid output gate in its gated-delta-net layers where Qwen3.6 uses silu. Converted progressively, three GDN layers at a time from the back, each group re-fitted then trained, then a settle pass. One-shot conversion put CE at 14.7 (dead); progressive landed at 2.05 held-out.
4. Hyper-connections. Four residual streams per layer plus a head mixer, retrofitted at an exact identity so the converted model was bit-for-bit unchanged at step 0, then trained. (Lesson banked: zero down and zero up is a saddle with no gradient — down random, up zero keeps identity and learns.)
5. N-gram memory. A 1.0B-parameter hashed table: 8 heads × 488,320 rows × 256, bigrams and trigrams hashed exactly the way llama.cpp's PLE does it, read into the residual before layer 2 through a gated key/value path. Trained with normalised sparse SGD on the host (each visited row moves a fixed 0.02 in its gradient direction — plain clipped SGD left the fp16 table frozen below the ulp, which is why the table is zero-initialised and this rule exists). It grows without retraining: doubling rows by replication (hash % 2R ∈ {r, r+R}) is behaviour-preserving, so the 27B is this table doubled and trained further.
6. Online distillation from Qwen3.8-27B. No cached logits. On corpus windows: forward KL over the full 248k vocab plus CE. On the student's own sampled replies: reverse KL over the full vocab at every reply token, graded live by the 27B (the "dark knowledge" of a stronger sibling with the same tokenizer). The on-policy gradient is accumulated into the next corpus step rather than stepped alone — stepping alone was 32× noisier and drifted.
7. Export to the stock format. HC norms folded the way the converter expects, inert QSA indexer (budget ≥ context), PLE tensors and hash contract emitted, model_type: qwen4_exp_text. convert_hf_to_gguf.py from stock llama.cpp, served by a stock CUDA build with 0 modified files.

Which weights are which

• Whittle-Next-26B-A3B-Q8_0.gguf — the model, stock llama.cpp.
• eval/gsm8k_200_stock_llamacpp_q8_0.json — the 200-question run, every answer. (The folder also holds eval/gsm8k_200_stock_llamacpp_q8_0_max1024.json and eval/gsm8k_200_stock_llamacpp_q8_0_max512_rerun.json.)
• train/hc_ple_step1000.pt, train/ple_hash.json, train/ngram_table_1B_fp16.npy — the trained HC/PLE/gates/routers, the hash contract and the raw table, so the 27B (or your own) can be grown from here.
• train/logs/ — the real logs, including the two on-policy attempts that were stopped and why.
• bf16/ — the full assembled qwen4_exp weights in bf16 (50 GB). base-sigmoid/ — the frozen body they were built on (47 GB).
• train/27b*/ — checkpoints, tables and logs from the 27B line's training phases on this body (27b, 27b-anchor, 27b-code, 27b-gentle, 27b-rounded, 27b-ship, 27b-tablepull). The Whittle-Next-27B-A3B card points here for the logs of the phases it tried and did not ship.

Train it further

Everything needed to continue training is in this repo — that is the point of publishing the pieces, not just the GGUF:

• base-sigmoid/ — the sigmoid-gate-converted Qwen3.6-Whittle-25B-A3B in bf16 (qwen3_5_moe layout, gate_act.json says which layers are sigmoid). This is the frozen body; load it with linear_attn.norm.activation = "sigmoid".
• train/hc_ple_step1000.pt — the trainable set: hyper-connections for all 40 layers + head mixer, PLE projections, shared-expert gates, routers. train/ple_hash.json — the hash contract (multipliers, rows, heads, EOS reset).
• train/ngram_table_1B_fp16.npy — the raw table (rows × 256, fp16 memmap). Grow it with grow_table.py (FACTOR=2 doubles rows by replication; outputs are bit-identical at the swap).

Trainer: logic65/mini-next-a100-kit/colab/train_next36.py (+ qwen4exp_modules.py). Point MODEL at base-sigmoid/, RESUME_CKPT at the checkpoint, TABLE/OUT at the table and hash. It trains only the pieces above (Muon for 2-D, AdamW for 1-D, normalised sparse SGD for the table), takes any qwen3_5/qwen3_5_moe HF model as an online teacher, and export_next36.py writes a stock-convertible qwen4_exp_text checkpoint. ~74 GB VRAM with the 27B teacher resident in int8; smaller teachers fit smaller cards. bf16/ holds the assembled qwen4_exp wrap (plain bf16 weights, 50 GB, 13 shards): exactly what convert_hf_to_gguf.py was run on, for anyone who wants the full weights rather than the pieces.

Run it

llama-server -m Whittle-Next-26B-A3B-Q8_0.gguf -ngl 99 -c 8192 --jinja -fa on

Request body: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05, chat_template_kwargs: {"enable_thinking": false}. Sample, don't decode greedily — greedy decoding is the single most reliable way to make any model in this family loop.

Caveats: what to expect

Conversational, follows the template, stops. Facts and maths at the base model's level (the GSM8K gain is within noise). The one visible style change is the teacher's: it likes to explain arithmetic step by step. The n-gram table is one hour old; its value grows with the overnight run, which is why the 27B exists.

Support this work

Whittle runs on one hobbyist's grocery budget and rented GPU hours. If this research is useful to you: ko-fi.com/davida81328 ☕

Authors

David Aylward (logic65) & Claude (Anthropic) — designed, debugged and verified together, one rented RTX PRO 6000 Blackwell, one day.

Provenance

Body: Qwen/Qwen3.6-35B-A3B. Teacher: Qwen/Qwen3.8-27B. Format: Qwen3.8-Flash-Next (qwen4_exp) as implemented in llama.cpp. All Apache-2.0. Method references: REAP (arXiv 2510.13999), "How to Score Experts for One-Shot MoE Expert Pruning" (arXiv 2606.15716), hyper-connections (arXiv 2409.19606), DistiLLM-2 / on-policy distillation literature. Scripts: logic65/mini-next-a100-kit/colab/ (prune_qwen36.py, heal_qwen36.py, train_next36.py, grow_table.py, export_next36.py). Part of the Whittle project by logic65.