Whittle-Qwen-3.8-35B-A3B — GGUF

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

Ready-to-run llama.cpp quantisations of logic65/Whittle-Qwen-3.8-35B-A3B: a 35.1 B-total, ~3 B-active mixture-of-experts in the Qwen3.8-Flash-Next (qwen4_exp) format with a 10 B-parameter n-gram memory that is load-bearing (the body measurably needs it; see the base card). Distilled from Qwen3.8-27B at the logit level on 25k verified teacher traces; built on Whittle-Next-27B-A3B. Runs on stock llama.cpp, no patches. The full card, measurements and caveats are on the base repo; this repo is the download.

Status

Current. These files are the Phase-2 step 32010 root (p2-s32010, 30 Sep 2026) of the current Whittle flagship: the complete Phase-2 distillation from Qwen3.8-27B (every usable teacher row seen once), a memory phase, and a 410-step fix for the llama.cpp web page's browser tools. The previous (p2-s6000) ladder was replaced in place; the earlier full weights stay under bf16-s6000/, bf16-agentfix2/, bf16-lw5/, bf16-lw2/ and bf16-tbl1/ on the base repo.

Which file

The memory is ~10.5 GB of each file at Q8 (proportionally less at lower bits). It is a lookup, one row per token per head, so keep it in system RAM with -ot per_layer_token_embd=CPU — the GPU then holds a 27 B-class footprint and generation runs at 3 B-active speed.

K-quants were requantised from the Q8_0. Every file carries the updated chat template, which accepts a system message anywhere in the conversation, so Claude Code works (it sends its environment block after the user turn). Serve the table whole: the body depends on the memory (see the base card); a build that drops or re-hashes per_layer_token_embd behaves like the v4.4 parent minus its knowledge.

Measured (summary; the full tables are on the base card)

Served as the Q8_0 it scores 23/24 on the stop/loop battery, 43/50 on unseen GSM8K, 46/60 on our MATH probe, 72/72 exact on a strict-JSON hold-out and 96/96 on our tool-use probes, and it finishes 31/36 held-out chats on the llama.cpp web page with its browser tools on (the step before the fix: 15/36); see the base card.

Run it

llama-server -m Whittle-Qwen-3.8-35B-A3B-Q5_K_M.gguf -ngl 99 -c 16384 --jinja -fa on -ot per_layer_token_embd=CPU

• sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05 — sample, do not decode greedily; greedy decoding loops on this family.
• thinking: "chat_template_kwargs": {"enable_thinking": true} — it does its best work with thinking on (distilled on complete thinking traces). It thinks longer than earlier checkpoints: give it max_tokens 4096+ for code and 8192+ for maths. --reasoning-format deepseek separates the thinking into reasoning_content.
• To also move the routed experts to RAM on small cards: -ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU".
• On llama.cpp builds that have --lazy-mode (newer than ~30 Aug 2026), add --lazy-mode off so the memory table is loaded into RAM; otherwise it is read lazily from disk and -ot is ignored, which makes prompt processing slow.
• Tool use / agents: keep each assistant turn's reasoning_content in the history for the rest of the tool episode.
• Architecture qwen4exp; if your build reports an unknown architecture, update llama.cpp.

Provenance

David Aylward (logic65) & Claude (Anthropic). Parent: logic65/Whittle-Next-27B-A3B. Teacher: Qwen/Qwen3.8-27B. Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0.