Whittle-Next 19.8B-A11B (chat) — router-trained

The trained, conversational lineage of Whittle-Next, carved from Whittle-tri-14.7B. 19.775B total parameters, ~11.5B active at the recommended k=104 (the -base repo's "A10B" is the figure at the trained k=58 — raising k activates 1.45B more of the routed experts).

⚠️ Research artifact. It holds a conversation, follows the chat template, writes fenced code and stops cleanly. It is also factually unreliable and its arithmetic is approximate. Treat it as an architecture demonstrator, not an assistant.

If this research is useful to you: ko-fi.com/davida81328 ☕

What is trained here, on top of the base carve

component	params	state
Routers (`mlp.gate`)	39.3M	trained jointly, at k=104
Shared-expert gates	164k	trained (own optimiser group, 10× LR)
Hyper-connections (4 residual streams)	988M	trained (woken from Eq-14 identity)
n-gram (PLE) projections	—	trained
4B n-gram memory table	4.0B	frozen, 6.25M rows × 640
240 experts, attention/GDN	7.55B / 4.7B	frozen (attention baked from an earlier A100 run)

Training routers alone was harmful in three earlier experiments. Trained jointly with the gates, hyper-connections and n-gram projections — and at the k they serve — they produced the best offline metrics this project has recorded: held-out CE 4.1466 → 3.9745 and the fact battery 4/5 → 5/5 (first perfect score).

Serving — these settings are REQUIRED

llama-server -m <gguf> -ngl 99 -c 8192 --jinja \
  --override-kv qwen4exp.expert_used_count=int:104

Request body: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05.

Never greedy. At temperature 0 a paragraph-length answer degenerates into verbatim repetition (4-gram repetition 0.885, never stops). With the sampler above the same prompt scores 0.000 and ends cleanly. This is the single largest cause of looping.
k=104. The router was trained at this k and is only valid at it.
Reasoning is verbose (a family trait of this Qwen lineage, inherited from the parent). With thinking ON, allow ≥700 tokens or the answer comes back empty — llama.cpp puts the unclosed block in reasoning_content and leaves content blank. For short factual turns pass chat_template_kwargs: {"enable_thinking": false}.

GGUF builds live at logic65/whittle-next (whittle-next-qwen4exp-router104-PLE4B-Q4_K_M.gguf is this lineage). A llama.cpp with qwen4exp support is required; the config declares output_gate_type: silu because these Qwen3.5-derived GDN weights need a SiLU output gate rather than the sigmoid a Flash-Next model uses.

How conversational is it, really?

Mechanically conversational; not yet substantively reliable. It takes a turn, answers, and stops — and the content underneath is often wrong.

probe	what it actually returned
"What is the capital of France?"	"The capital of France is Paris." ✅
"Write a minimal HTML page with a heading"	a complete, valid, fenced HTML page ✅
"Write a short paragraph about the ocean"	coherent paragraph, zero repetition, clean stop ✅
"Explain in two sentences why the sky is blue"	"sunlight shines through the clouds and makes it so bright" ❌ confidently wrong
"What is 17 + 25?"	32 (also 20 or 49 in other modes) ❌
"List exactly 5 fruits, numbered 1-5"	sometimes "1, 2, 3, 4, and 5 are fruits" ❌ hears the format, misses the substance

Untested: every probe above is single-turn. Whether it holds context across several exchanges — arguably the real test of "conversational" — has not been measured, and we make no claim about it.

The failure mode has moved from broken generation to a small model with damaged knowledge. Closing that gap is what the n-gram memory work targets, not further architecture changes.

Honest limitations

Facts are unreliable — "why is the sky blue" gets a confident wrong answer. The n-gram memory measurably helps perplexity (86.6 → 33.2 on wikitext-2 in-domain) but does not make it knowledgeable.
Arithmetic is approximate: 17 + 25 returns 32, 20 or 49 depending on mode. It estimates where the parent computed.
It invents URLs and citations rather than declining. Do not trust a link it produces.
It over-thinks. Reasoning mode spends thousands of characters on trivial questions.
Long code generations can decay under greedy decoding; use the sampler.

Provenance

Qwen3.8-27B → depth-compressed by parallel-compose merging (64 layers → 32) to Whittle-tri-14.7B, FFN width left UNCUT at 17408 → carved to a 240-expert qwen4exp MoE (240 x 64 + 2048 shared = 17408 exactly) with hyper-connections and n-gram memory → SFT + joint router/gate training on 2× free Kaggle T4s.