Whittle-Next-27B-A3B — GGUF

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

Ready-to-run llama.cpp quantisations of logic65/Whittle-Next-27B-A3B: a 27B-total, ~3B-active mixture-of-experts in the Qwen3.8-Flash-Next (qwen4_exp) format, with hyper-connection residual streams and a 2B-row hashed n-gram memory transferred from Qwen3.8-Flash-Next, distilled from Qwen3.8-27B. Runs on stock llama.cpp, no patches. The full model card, training history, caveats and measurements live on the base repo; this repo is the download.

Status

v4.4 ladder of Whittle-Next-27B-A3B. The line has since continued as Whittle-Qwen-3.8-35B-A3B (this body plus a 10 B n-gram memory), whose GGUFs are on Whittle-Qwen-3.8-35B-A3B-GGUF. The compute budget for this line is spent; the full distillation it still needs is described on the base card.

This is a research preview, not a finished general-purpose model: a working body for anyone who wants to build on the Qwen4-Next format, with every piece needed to keep training it published alongside. Maths sits at the level of the 26B predecessor (GSM8K ~86%), it stops cleanly (12/12 on the stop probe), and code answers are correct but long. Read Caveats, measured on the base repo before relying on it.

Which file

v4.4 (colab1 step4400, the root weights of the base repo): everything unlocked and distilled on 1,840 complete Qwen3.8-27B maths and code-review traces; measurably closer to the teacher (layer parity 83.4% → 87.4% on unseen rows), maths probe 48/60 vs 43/60, same run instructions. v4.3 lives on as bf16-v4.3/ on the base repo.

Only ~3B parameters are active per token, so generation speed is that of a 3B model at the memory footprint above; the 2B-row n-gram memory is part of the file and is looked up, not computed. K-quants were quantised from the bf16 GGUF of the root weights.

Run it

llama-server -m Whittle-Next-27B-A3B-v4.4-Q5_K_M.gguf -ngl 99 -c 16384 --jinja -fa on

Then POST to /v1/chat/completions with:

• sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05 — sample, do not decode greedily; greedy decoding loops on this family.
• thinking: "chat_template_kwargs": {"enable_thinking": true} — the model does its best work with thinking on (it was distilled on complete thinking traces). Use false for terse direct answers.
• budget: max_tokens of 4096 or more for code; with thinking on, a short budget returns an empty answer because it all goes to thinking.
• --reasoning-format deepseek on the server separates the thinking into reasoning_content.

Works with anything that embeds a recent llama.cpp (LM Studio, Jan, Ollama via a Modelfile pointing at the file, llama-swap). The architecture is qwen4exp; if your build reports an unknown architecture, update llama.cpp.

Less VRAM: keep the n-gram memory (and, if needed, the experts) in system RAM

The 2B-row memory is a single lookup tensor, per_layer_token_embd.weight (2.1 GB at Q8). It is read one row per token per head, so serving it from system RAM costs almost nothing. Add:

-ot "per_layer_token_embd=CPU"

That takes ~2 GB off the card (Q5_K_M then fits a 16 GB card with room for context). For cards smaller than that, also move the routed experts to RAM; generation then runs at CPU-expert speed (about 4-8 tok/s on a desktop, prefill stays on the GPU):

-ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU"

Attention, hyper-connections, the shared expert and the memory's key/value projections stay on the GPU in both cases.

Reasoning format: Qwen3.8's

The thinking style is inherited, not just enabled. v3 to v4.4 were distilled on roughly 1,500 complete Qwen3.8-27B thinking traces with the teacher's next-token distributions at every position, and the training data was filtered to traces that follow the format exactly (one <think> block, closed, then the answer, then a single end-of-turn token). The student reproduces it: in the 40-prompt thinking-on probe every reply that finished within budget closed its think block and stopped on EOS, and the planning register inside the block is the 27B's. In practice this means anything built for Qwen3.8 output works unchanged: --reasoning-format deepseek separates the block into reasoning_content, the <think>/</think> tags are the same tokens, and enable_thinking: false yields the same terse direct-answer mode.

Authors and provenance

David Aylward (logic65) & Claude (Anthropic). Body: Qwen/Qwen3.6-35B-A3B. Teacher: Qwen/Qwen3.8-27B. Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0. Part of the Whittle project by logic65.