Qwen3.8-Whittle-Next — 19.8B, ~10B active

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

A Qwen3.8-Flash-Next-style architecture (qwen4_exp) built by surgery instead of pretraining: a training-free MoE carve of a compressed 14.7B dense model, wrapped in hyper-connections, with a 4-billion-parameter n-gram memory bolted on — running end-to-end in llama.cpp.

The table is the point: capacity bought with memory instead of FLOPs. It adds 4B parameters and a few gathered rows per token.

Status

Experimental research artefact — not an assistant; superseded by Whittle-Next-27B-A3B (the parent of the current Whittle-Qwen-3.8-35B-A3B). This is the record of an experiment, published so it can be reproduced, not deployed. It fails ~80% of a standard conversational loop test. It has had no completed instruction tuning and no safety work. Use it to study the architecture; do not put it in front of users.

Which weights are which

• Root (model-00001.safetensors … model-00009.safetensors, config, tokenizer): the checkpoint this card describes. SFT (ultrachat multi-turn + reasoning traces) was in progress when the GPU window closed; the checkpoint here carries its best gate-selected state.
• wake/ (same layout, 9 shards): the hyper-connection mix path, woken (added the same day as the root). After every earlier run, the HC low-rank mix was measured dead at exactly zero (a mutual gradient deadlock in the identity init - see finding 5 below). wake/ carries the first checkpoint trained with it alive, in tandem with chat SFT: the 5-probe factual battery reached 5/5 (the gold-symbol and largest-planet probes pass for the first time), greedy 4-gram repetition 0.056. Stopping (EOS) remains weak - that is the top of the roadmap. Load with subfolder="wake", or use the wakeHC GGUF.
• GGUFs: Qwen3.8-Whittle-Next-19.8B-A10B-base-Q4_K_M.gguf (12.31 GiB), the Q4_K_M build published with the root checkpoint, and Qwen3.8-Whittle-Next-19.8B-A10B-base-wakeHC-Q4_K_M.gguf (12.31 GiB), the Q4_K_M build of wake/.
• whittle.svg.

Measured (wikitext-2 test, ctx 512, 24 chunks, same session)

Read the next line before quoting that number. On science text the table never saw, n-gram on vs off is 27.53 vs 27.96 — a 1.5% gain. The in-domain 55% drop is mostly memorisation of the training corpus, not generalised capacity: this table has so far seen ~2% of one epoch of its mixed corpus. A memory is only as good as what has been written into it — that is both the honest caveat and the design thesis.

The n-gram does not worsen degeneration: on the 94-generation loop test it improved failures 86→79 and looping 42→31 versus the same model with the table off.

How it was made

Qwen3.8-27B → Whittle-tri-14.7B (layer-merge compression) → ExpertWeaver-style carve + DOT-MoE alignment (arXiv 2602.15521, 2606.01666) → HC/PLE retrofit (this repo). The n-gram table and adapters saw: wikitext-2, fineweb-edu, cosmopedia/openstax, open-web-math, wikihow, and fenced code (code_x_glue, Magicoder-OSS, CodeFeedback) — a few million tokens total, far short of the corpus.

Every training script, log, probe and patch: logic65/mini-next-a100-kit (private at the time of writing).

What this repo taught us (the findings are the product)

1. llama.cpp's qwen4exp GDN uses a sigmoid output gate; Qwen3.5-derived GDN weights need silu. No weight transform can bridge them (silu is unbounded). One-line patch included (patches/ in the kit). Perplexity with sigmoid: 36,138 — with silu: 85.02.
2. The converter folds +1 into hc_norm.weight (Gemma-style zero-centred gammas), so an identity retrofit must store 2(1+w)−1. Storing 2(1+w) costs a 16× perplexity blow-up.
3. heads_per_ngram is per gram order — total heads = (ngram−1) × heads_per_ngram.
4. The PLE hash, verified bit-for-bit: one shared multiplier vector, mixed = Σ⊕ ctx[j]·m[j] per order; heads differ only by % vocab + offset; EOS resets the window. Verified by a probe table whose rows spell out their own indices through llama-eval-callback — 8/8 exact.
5. Two init deadlocks that silently disable training. The PLE returned only its conv path (llama.cpp returns hidden + gated + conv), so with zero init nothing had gradient — the first 4B table "trained" overnight and moved 3.6% of rows, all numerical noise. And the HC identity init (down=0, up=0) mutually kills both gradients: after every run, all 64 mix modules still had |down|=|up|=0.000 exactly. The 845M-param mix path never learned — only hc_norm and inject did. Fix: init up~N(0,0.02), down=0 — still exactly identity, gradient alive from step 1.
6. A hand-written training loop gets no gradient checkpointing from HF. The 4-stream residual made activations ~40 GB of a 79 GB card; checkpointing each layer body let 3× more parameters train in less memory (64.9 GB) at 2× step time.

Run it

GGUF (recommended): the .gguf files here run on llama.cpp with the one-line silu-gate patch (patches/qwen4exp-silu-gate.patch in the kit, private at the time of writing). Upstreaming a gate flag is planned. HF transformers loading of this text-only qwen4_exp export is untested.

Caveats

• Chat remains broken. That is the top of the roadmap, not a surprise.
• The in-domain perplexity gain from the n-gram table is mostly memorisation of the training corpus (see Measured).

Provenance

Built by David Aylward with Claude (Anthropic) as co-author. Thanks to the contributor who pointed at free weekly T4 quota — that is where the next epochs run. Every experiment on this card was paid for out of pocket. Lineage and licences: Qwen3.8-27B → Whittle-tri-14.7B → this repo; Apache-2.0.

☕ ko-fi.com/davida81328 — GPU hours, not coffee.