Whittle 14.7B logit-distillation adapter (research preview)
☕ Support this work
Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.
LoRA r32 trained to match the output distribution of the Qwen3.8-27B parent, not just the reference answers. Targets are the parent's top-128 log-probs per token (recorded once over the heal corpus), and the loss is KL against those plus a small cross-entropy anchor, taken on assistant turns only.
Status
Earlier line (Aug 2026); the current Whittle models are the Whittle-Next / Whittle-Qwen-3.8 line: Whittle-Qwen-3.8-35B-A3B.
Research preview, needs further work. Published as a record of method and measurements. See the Whittle collections for the models themselves: models you can run and research and dev.
What is in this repo
adapter_config.json,adapter_model.safetensors(446 MB): the LoRA r32 adapter (peft format) for logic65/Qwen3.8-Whittle-tri-14.7B.kd_loss_curve.png,kd_loss_log.csv: the training loss curve and its log.chat_template.jinja,generation_config.json,tokenizer.json,tokenizer_config.json.
How it was made
Why distribution rather than labels: the base model already predicts this corpus almost perfectly (cross-entropy ~4e-05) while its distribution sits ~2.5 nats from the parent's. Hard labels had nothing left to teach; the distribution did.
3.90M tokens, ~1.2 epochs, 80 minutes on one A100. The teacher pass ran the parent at nf4, so the deep tail of the targets is approximate.
Provenance and licence
Adapter for logic65/Qwen3.8-Whittle-tri-14.7B, a compression of Qwen3.8-27B; the distillation teacher is the Qwen3.8-27B parent. Apache-2.0.