Qwen3.5-9B Kimi-k3 Distilled — oQ6e-mtp (MLX, Apple Silicon)
MLX build of khazarai/Qwen3.5-9B-Kimi-k3-Distilled,
quantized from the BF16 source with oMLX oQ6e (imatrix-calibrated 6-bit), MTP head kept.
A small agentic-coding model distilled from Kimi K3 traces; until now it had no MLX/oQ build.
What this build is
- Model: 9B dense Qwen3.5 (hybrid Gated DeltaNet + attention, 32 layers), fine-tuned by khazarai on Kimi K3 coding/debugging agent trajectories (tool-first, build-test-fix loops). 262k context.
- Quantization: oMLX oQ6e — 6-bit affine (group 64) with sensitive tensors at 8-bit, imatrix-calibrated on oMLX's code + multilingual set. Effective 7.05 bits/weight for the full checkpoint.
- MTP head kept (
-mtp): the source's multi-token-prediction layer is preserved, so oMLX can use it for speculative decoding. - Vision encoder kept from the source (image input not tested in this build).
- Size: ~8.5 GB (source BF16: ~19 GB). Other sizes:
oQ8e-mtp(~10 GB). - Runtime: requires oMLX ≥ 0.7.1. Not tested with plain
mlx-lm/ LM Studio.
Quantization and bit distribution
| Storage format | Logical weights | Share of total | Tensor storage | Effective bits/weight |
|---|---|---|---|---|
| Affine 6-bit, group 64 | 8.845B | 91.631% | 6.693 GiB | 6.50 |
| Affine 8-bit, group 64 | 0.317B | 3.286% | 0.314 GiB | 8.50 |
| BF16 | 0.491B | 5.083% | 0.914 GiB | 16.00 |
| Total | 9.653B | 100% | 7.921 GiB | 7.049 |
| Component | Logical weights | Storage |
|---|---|---|
| Language model | 8.954B | 7.356 GB / 6.850 GiB |
| Vision | 0.456B | 0.912 GB / 0.849 GiB |
| MTP head | 0.243B | 0.238 GB / 0.221 GiB |
| Total | 9.653B | 8.505 GB / 7.921 GiB |
Quality vs. BF16 source
Full-vocabulary KL against the BF16 source, same token streams; 64 sequences × 512 tokens (32.7k scored tokens): code, English prose, Turkish.
| Metric | Value |
|---|---|
| KL(BF16 ‖ oQ6e), mean | 0.00315 nat/token |
| KL median / p99 | 0.0022 / 0.019 |
| Top-1 token agreement | 97.0 % |
| Perplexity BF16 → oQ6e | 8.428 → 8.445 |
No benchmark suite was run on this build; quality is reported as divergence from the source.
Speed (Mac Studio M3 Ultra, 512 GB, oMLX 0.7.1)
| Prefill, ~29k-token prompt (cold) | ~21 s (~1,370 tok/s) |
| Same prefix again (prefix cache) | ~0.54 s |
| Decode (MTP speculative decoding) | ~87 tok/s |
Tool calling (Anthropic /v1/messages / OpenAI /v1/chat/completions, 5 runs each) |
4/5 / 5/5 |
Single-run measurements; numbers vary with context length, content and settings (MTP gains depend on how predictable the output is).
Usage (oMLX)
Place the folder in your oMLX model directory and restart oMLX; it is served on both the OpenAI and Anthropic APIs:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen3.5-9B-Kimi-k3-Distilled-oQ6e-mtp", "messages": [{"role": "user", "content": "Write a Python function that parses a CSV line."}]}'
Memory: ~8.5 GB weights plus KV cache (~32 KB/token on the 8 full-attention layers → ~8 GB at 262k).
Credits & license
- Base model: Qwen team — Qwen3.5-9B (via
unsloth/Qwen3.5-9B). - Kimi K3 distillation fine-tune: khazarai —
khazarai/Qwen3.5-9B-Kimi-k3-Distilled(datasetgreghavens/kimi-k3-coding-and-debugging-traces). - Quantization: oMLX oQ.
- License: Apache-2.0, inherited from upstream.