Sign up

logic65/Whittle-Qwen-3.8-35B-A3B-GGUF

🤗 Hugging Face sourcetext-generationapache-2.0261 GBGGUF✓ HF checksums availableupdated today
No torrent yet →

Whittle-Qwen-3.8-35B-A3B — GGUF

Ready-to-run llama.cpp quantisations of logic65/Whittle-Qwen-3.8-35B-A3B: a 35.1 B-total, ~3 B-active mixture-of-experts in the Qwen3.8-Flash-Next (qwen4_exp) format with a 10 B-parameter n-gram memory that is load-bearing (zeroing it costs +2.12 nats on unseen code). Distilled from Qwen3.8-27B; built on Whittle-Next-27B-A3B. Runs on stock llama.cpp, no patches. The full card, measurements and caveats are on the base repo; this repo is the download.

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If these weights are useful to you, or you want to see the memory grow further: ko-fi.com/davida81328. Every hour of GPU time goes into the next checkpoint.

Which file

The memory is ~10.5 GB of each file at Q8 (proportionally less at lower bits). It is a lookup, one row per token per head, so keep it in system RAM with -ot per_layer_token_embd=CPU — the GPU then holds a 27 B-class footprint and generation runs at 3 B-active speed.

file size notes
Whittle-Qwen-3.8-35B-A3B-Q8_0.gguf 37.8 GB reference quant; the numbers on the base card were measured on the full weights it was made from
Whittle-Qwen-3.8-35B-A3B-Q6_K.gguf 29.3 GB near-lossless
Whittle-Qwen-3.8-35B-A3B-Q5_K_M.gguf 25.1 GB recommended for one 24 GB card with the memory in RAM
Whittle-Qwen-3.8-35B-A3B-Q4_K_M.gguf 21.3 GB good default for 16–20 GB cards, memory in RAM
Whittle-Qwen-3.8-35B-A3B-Q3_K_M.gguf 16.7 GB smallest; expect some loss on maths

These files are the lw2 root (19 Sep 2026, evening); the earlier tbl1 GGUFs were replaced in place — the tbl1 full weights stay under bf16-tbl1/ on the base repo. K-quants were requantised from the Q8_0. Serve the table whole: the body depends on the memory (see the base card); a build that drops or re-hashes per_layer_token_embd behaves like the v4.4 parent minus its knowledge.

Run it

llama-server -m Whittle-Qwen-3.8-35B-A3B-Q5_K_M.gguf -ngl 99 -c 16384 --jinja -fa on -ot per_layer_token_embd=CPU
  • sampler: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05 — sample, do not decode greedily; greedy decoding loops on this family.
  • thinking: "chat_template_kwargs": {"enable_thinking": true} — it does its best work with thinking on (distilled on complete thinking traces). Give it max_tokens 4096+ for code. --reasoning-format deepseek separates the thinking into reasoning_content.
  • To also move the routed experts to RAM on small cards: -ot "per_layer_token_embd=CPU" -ot "\.ffn_(up|down|gate)_exps\.=CPU".
  • Architecture qwen4exp; if your build reports an unknown architecture, update llama.cpp.

Provenance

David Aylward (logic65) & Claude (Anthropic). Parent: logic65/Whittle-Next-27B-A3B. Teacher: Qwen/Qwen3.8-27B. Memory contents: Qwen/Qwen3.8-Flash-Next. All Apache-2.0.