Qwen3.5-9B Kimi-k3 Distilled — oQ8e-mtp (MLX, Apple Silicon)
MLX build of khazarai/Qwen3.5-9B-Kimi-k3-Distilled,
quantized from the BF16 source with oMLX oQ8e (imatrix-calibrated 8-bit), MTP head kept.
A small agentic-coding model distilled from Kimi K3 traces; until now it had no MLX/oQ build.
What this build is
- Model: 9B dense Qwen3.5 (hybrid Gated DeltaNet + attention, 32 layers), fine-tuned by khazarai on Kimi K3 coding/debugging agent trajectories (tool-first, build-test-fix loops). 262k context.
- Quantization: oMLX oQ8e — 8-bit affine (group 64), imatrix-calibrated on oMLX's code + multilingual set.
- MTP head kept (
-mtp): the source's multi-token-prediction layer is preserved, so oMLX can use it for speculative decoding. - Vision encoder kept from the source (image input not tested in this build).
- Size: ~10 GB (source BF16: ~19 GB). Other sizes:
oQ6e-mtp(~8 GB). - Runtime: requires oMLX ≥ 0.7.1. Not tested with plain
mlx-lm/ LM Studio.
Quality vs. BF16 source
Full-vocabulary KL against the BF16 source, same token streams; 64 sequences × 512 tokens (32.7k scored tokens): code, English prose, Turkish.
| Metric | Value |
|---|---|
| KL(BF16 ‖ oQ8e), mean | 0.00097 nat/token |
| KL median / p99 | 0.00065 / 0.0054 |
| Top-1 token agreement | 98.3 % |
| Perplexity BF16 → oQ8e | 8.428 → 8.428 |
No benchmark suite was run on this build; quality is reported as divergence from the source.
Speed (Mac Studio M3 Ultra, 512 GB, oMLX 0.7.1)
| Prefill, ~29k-token prompt (cold) | ~24 s (~1,200 tok/s) |
| Same prefix again (prefix cache) | ~0.58 s |
| Decode (MTP speculative decoding) | ~70 tok/s |
Tool calling (Anthropic /v1/messages / OpenAI /v1/chat/completions, 5 runs each) |
5/5 / 4/5 |
Single-run measurements; numbers vary with context length, content and settings (MTP gains depend on how predictable the output is).
Usage (oMLX)
Place the folder in your oMLX model directory and restart oMLX; it is served on both the OpenAI and Anthropic APIs:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen3.5-9B-Kimi-k3-Distilled-oQ8e-mtp", "messages": [{"role": "user", "content": "Write a Python function that parses a CSV line."}]}'
Memory: ~10 GB weights plus KV cache (~32 KB/token on the 8 full-attention layers → ~8 GB at 262k).
Credits & license
- Base model: Qwen team — Qwen3.5-9B (via
unsloth/Qwen3.5-9B). - Kimi K3 distillation fine-tune: khazarai —
khazarai/Qwen3.5-9B-Kimi-k3-Distilled(datasetgreghavens/kimi-k3-coding-and-debugging-traces). - Quantization: oMLX oQ.
- License: Apache-2.0, inherited from upstream.