Whittle-Next 19.8B-A11B (chat) — router-trained
☕ Support this work
Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.
The trained, conversational lineage of Whittle-Next,
carved from Whittle-tri-14.7B.
19.775B total parameters, ~11.5B active at the recommended k=104 (the -base repo's "A10B"
is the figure at the trained k=58 — raising k activates 1.45B more of the routed experts).
⚠️ Research artifact. It holds a conversation, follows the chat template, writes fenced code and stops cleanly. It is also factually unreliable and its arithmetic is approximate. Treat it as an architecture demonstrator, not an assistant.
Status
Earlier Whittle-Next release (research artefact). The Whittle-Next line continued with Whittle-Next-27B-A3B, parent of the current Whittle-Qwen-3.8-35B-A3B.
Which weights are which
- This repo: the router-trained checkpoint as safetensors (
model-00001.safetensors…model-00009.safetensors, config, tokenizer) andwhittle.svg. - GGUF builds live at logic65/whittle-next
(
whittle-next-qwen4exp-router104-PLE4B-Q4_K_M.ggufis this lineage).
Measured
How conversational is it, really?
Mechanically conversational; not yet substantively reliable. It takes a turn, answers, and stops — and the content underneath is often wrong.
| probe | what it actually returned |
|---|---|
| "What is the capital of France?" | "The capital of France is Paris." ✅ |
| "Write a minimal HTML page with a heading" | a complete, valid, fenced HTML page ✅ |
| "Write a short paragraph about the ocean" | coherent paragraph, zero repetition, clean stop ✅ |
| "Explain in two sentences why the sky is blue" | "sunlight shines through the clouds and makes it so bright" ❌ confidently wrong |
| "What is 17 + 25?" | 32 (also 20 or 49 in other modes) ❌ |
| "List exactly 5 fruits, numbered 1-5" | sometimes "1, 2, 3, 4, and 5 are fruits" ❌ hears the format, misses the substance |
Untested: every probe above is single-turn. Whether it holds context across several exchanges — arguably the real test of "conversational" — has not been measured, and we make no claim about it.
The failure mode has moved from broken generation to a small model with damaged knowledge. Closing that gap is what the n-gram memory work targets, not further architecture changes.
Offline metrics of the joint router training
Training routers alone was harmful in three earlier experiments. Trained jointly with the gates, hyper-connections and n-gram projections — and at the k they serve — they produced the best offline metrics this project has recorded: held-out CE 4.1466 → 3.9745 and the fact battery 4/5 → 5/5 (first perfect score).
How it was made
What is trained here, on top of the base carve
| component | params | state |
|---|---|---|
Routers (mlp.gate) |
39.3M | trained jointly, at k=104 |
| Shared-expert gates | 164k | trained (own optimiser group, 10× LR) |
| Hyper-connections (4 residual streams) | 988M | trained (woken from Eq-14 identity) |
| n-gram (PLE) projections | — | trained |
| 4B n-gram memory table | 4.0B | frozen, 6.25M rows × 640 |
| 240 experts, attention/GDN | 7.55B / 4.7B | frozen (attention baked from an earlier A100 run) |
Lineage: Qwen3.8-27B → depth-compressed by parallel-compose merging (64 layers → 32) to Whittle-tri-14.7B, FFN width left UNCUT at 17408 → carved to a 240-expert qwen4exp MoE (240 x 64 + 2048 shared = 17408 exactly) with hyper-connections and n-gram memory → SFT + joint router/gate training on 2× free Kaggle T4s.
Run it
Serving — these settings are REQUIRED:
llama-server -m <gguf> -ngl 99 -c 8192 --jinja \
--override-kv qwen4exp.expert_used_count=int:104
Request body: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05.
- Never greedy. At
temperature 0a paragraph-length answer degenerates into verbatim repetition (4-gram repetition 0.885, never stops). With the sampler above the same prompt scores 0.000 and ends cleanly. This is the single largest cause of looping. - k=104. The router was trained at this k and is only valid at it.
- Reasoning is verbose (a family trait of this Qwen lineage, inherited from the parent). With
thinking ON, allow ≥700 tokens or the answer comes back empty — llama.cpp puts the unclosed
block in
reasoning_contentand leavescontentblank. For short factual turns passchat_template_kwargs: {"enable_thinking": false}.
A llama.cpp with qwen4exp
support is required; the config declares output_gate_type: silu because these Qwen3.5-derived
GDN weights need a SiLU output gate rather than the sigmoid a Flash-Next model uses.
Caveats
- Facts are unreliable — "why is the sky blue" gets a confident wrong answer. The n-gram memory measurably helps perplexity (86.6 → 33.2 on wikitext-2 in-domain) but does not make it knowledgeable.
- Arithmetic is approximate: 17 + 25 returns 32, 20 or 49 depending on mode. It estimates where the parent computed.
- It invents URLs and citations rather than declining. Do not trust a link it produces.
- It over-thinks. Reasoning mode spends thousands of characters on trivial questions.
- Long code generations can decay under greedy decoding; use the sampler.
Provenance
David Aylward (logic65) & Claude (Anthropic) — designed, debugged and trained together. Apache-2.0.