Whittle-Qwen-3.8-35B-A3B — GGUF

Ready-to-run llama.cpp quantisations of logic65/Whittle-Qwen-3.8-35B-A3B: a 35.1 B-total, ~3 B-active mixture-of-experts in the Qwen3.8-Flash-Next (qwen4_exp) format with a 10 B-parameter n-gram memory that is load-bearing (the body measurably needs it; see the base card). Distilled from Qwen3.8-27B at the logit level on 25k verified teacher traces; built on Whittle-Next-27B-A3B. Runs on stock llama.cpp, no patches. The full card, measurements and caveats are on the base repo; this repo is the download.

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If these weights are useful to you, or you want to see the memory grow further: ko-fi.com/davida81328. Every hour of GPU time goes into the next checkpoint.

Which file

The memory is ~10.5 GB of each file at Q8 (proportionally less at lower bits). It is a lookup, one row per token per head, so keep it in system RAM with -ot per_layer_token_embd=CPU — the GPU then holds a 27 B-class footprint and generation runs at 3 B-active speed.

These files are the Phase-2 step 6000 root (p2-s6000, 28 Sep 2026): lw5 plus 6,000 steps of offline logit distillation from Qwen3.8-27B. Served as the Q8_0 it scores 24/24 on the stop/loop battery, 48/50 on unseen GSM8K, 44/60 on our MATH probe (48/60 with room to think) and 72/72 valid JSON on a strict-JSON hold-out, and 96/96 on our tool-use probes (stop after success, exact extraction); see Phase 2 on the base card. The previous (agentfix2) ladder was replaced in place; the earlier full weights stay under bf16-agentfix2/, bf16-lw5/, bf16-lw2/ and bf16-tbl1/ on the base repo. Every file carries the updated chat template, which accepts a system message anywhere in the conversation, so Claude Code works (it sends its environment block after the user turn). K-quants were requantised from the Q8_0. Serve the table whole: the body depends on the memory (see the base card); a build that drops or re-hashes per_layer_token_embd behaves like the v4.4 parent minus its knowledge.

Run it

llama-server -m Whittle-Qwen-3.8-35B-A3B-Q5_K_M.gguf -ngl 99 -c 16384 --jinja -fa on -ot per_layer_token_embd=CPU

• sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05 — sample, do not decode greedily; greedy decoding loops on this family.
• thinking: "chat_template_kwargs": {"enable_thinking": true} — it does its best work with thinking on (distilled on complete thinking traces). It thinks longer than earlier checkpoints: give it max_tokens 4096+ for code and 8192+ for maths. --reasoning-format deepseek separates the thinking into reasoning_content.
• To also move the routed experts to RAM on small cards: -ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU".
• Tool use / agents: keep each assistant turn's reasoning_content in the history for the rest of the tool episode.
• Architecture qwen4exp; if your build reports an unknown architecture, update llama.cpp.

Provenance

David Aylward (logic65) & Claude (Anthropic). Parent: logic65/Whittle-Next-27B-A3B. Teacher: Qwen/Qwen3.8-27B. Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0.