Whittle tri 14.7B chat (RESEARCH PREVIEW - REQUIRES FURTHER POST TRAINING)
A 14.7B compression of Qwen3.8-27B, repaired in two rounds so that it can hold a conversation. This is a research preview. It still needs further post training and its factual accuracy is weak. Published as an honest work in progress.
The problem
The compressed model kept its knowledge but could not talk: in open-ended chat it collapsed into repetition ("the fan fan fan..."), and asked for an iterative Fibonacci it wrote a broken recursive one. The cause was the training signal, not the weights. The original heal packed conversations into 256-token blocks and took the loss on every token, so predicting questions was rewarded as much as answering them.
Round 1 - instruction repair (SFT)
One conversation per sequence at 4096, never truncated mid-conversation, loss on assistant turns only. 4.58M tokens, 1.33 epochs, 90 minutes on one A100.
Round 2 - distillation from the parent (KD)
Top-64 logits cached from the Qwen3.8-27B parent, then a decoupled top-k objective: a binary KL matching the total probability mass the student places on the teacher's top-k set, plus a conditional KL matching the shape within that set, plus a CE anchor. Matching only the shape (the naive approach) leaves total set mass unconstrained, and the student drifts into over-confident, repetitive sampling. r=128, lr 5e-5, 55 min.
Measured, same prompts and sampler throughout
| test | base | after SFT | after SFT+KD |
|---|---|---|---|
| 4-gram repetition, 3-turn chat | 0.55 / 0.58 | 0.38 / 0.39 | 0.04 / 0.10 / 0.01 |
| instruction following (15 templated checks) | 11/15 | 11/15 | 11/15 |
| 39-prompt battery (raw completions) | 34/39 | 31/39 | 32/39 |
| "Fibonacci, iteratively" | recursive, broken | correct | correct |
KD recovered four raw-completion items that SFT had cost (add3, square, half,
py_print) without bringing the looping back: SFT fixes conversation, KD pulls the
distribution back toward the parent.
Known limitations
- Factual accuracy is poor. It states wrong things confidently.
- It sometimes returns an empty answer.
- Needs further post training. This is a starting point, not a finished model.
Files
whittle-tri-sftkd-q8_0.gguf- recommended, final SFT+KD model, Q8_0, 15.6GBwhittle-tri-sft-q8_0.gguf- round 1 only, kept for comparison- bf16 safetensors in the repo root (round 1; final bf16 on request)
arch qwen35, EOS 248046. Serve with thinking off. Samplers: temp 0.7, top_p 0.8, top_k 20.
Funding
Self funded on personal hardware and paid Colab time. If this is useful, support keeps it going: ko-fi.com/davida81328