This is a 4-bit MLX quantization of Jackrong/Kestrel-4B-Coder, a coding and agent-focused fine-tune of Qwen3.5-4B. It was produced with oMLX's oQ4e streaming quantizer and preserves both the vision tower and the integrated multi-token-prediction (MTP) head.
A sibling FP16 build is available as Kestrel-4B-Coder-oQ4e-fp16-mtp.
| Field | Value |
|---|---|
| Base model | Jackrong/Kestrel-4B-Coder @ 46fdfa2c75949c6e3abd4c62d7a1c079ffc18245 |
| Upstream lineage | Qwen/Qwen3.5-4B → unsloth/Qwen3.5-4B → Kestrel |
| Architecture | qwen3_5 VLM (Qwen3_5ForConditionalGeneration) |
| Quantization | oQ4e — oQ level 4, enhanced imatrix, group_size=64, affine |
| Storage dtype | bfloat16 (quantized weights packed as U32) |
| MTP | preserved — 1 layer, 29 tensors under language_model.mtp.* |
| Vision tower | preserved — 297 tensors, BF16, not quantized |
| Tensors | 1250 total (953 language + 297 vision); 0 lm_head |
| Size | 3,265,548,624 bytes (3.27 GB) |
| Imatrix | oqe_code_multilingual, 128 samples × 512 tokens, 256 entries |
| Runtime | oMLX ≥ 0.7.1 with MLX 0.32.4 |
quantize_oq_streaming(
model_path="Kestrel-4B-Coder",
output_path="Kestrel-4B-Coder-oQ4e-mtp",
oq_level=4, group_size=64,
text_only=False, dtype="bfloat16",
preserve_mtp=True, enhanced=True,
imatrix_num_samples=128, imatrix_seq_length=512,
auto_proxy_sensitivity=True, imatrix_reuse_cache=True,
stream_calibration=None,
)
text_only=False keeps the vision tower at source precision; preserve_mtp=True keeps the MTP head and its config fields. The imatrix cache was freshly collected for this build (cache_reused=false).
All tests below were run against this artifact after quantization, on 2026-10-26.
Environment: Apple M2 Max, 96 GiB unified memory; macOS (Darwin 25.6.0); oMLX 0.7.1; MLX 0.32.4; mlx-lm 0.31.4.dev132; mlx-vlm 0.7.8; transformers 5.17.0; Python 3.11.17.
| # | Test | Result | Measured |
|---|---|---|---|
| T1 | Source integrity | PASS | 738 tensors; both shard SHA-256 match the Hub (9.32 GB) |
| T2 | Pre-quant estimate | PASS | 5.61 bpw, 3.267 GB expected |
| T3 | Structural audit | PASS | 1250 tensors; 297 vision (all BF16, zero .scales/.biases); 29 MTP; 0 lm_head; U32 256 / BF16 994 |
| T4 | Config audit | PASS | mtp_num_hidden_layers=1 (root and text), unsloth_fixed_mtp=true, 4-bit/64/affine, 92 per-layer boosts |
| T5 | Patched load (mlx_vlm + oMLX patch) |
PASS | language_model.mtp attached with 1 layer; MTP norm matches on-disk tensor; vision 333.51M params |
| T6 | Text smoke, 5 coding prompts × {greedy, shipping} | PASS | Coherent, EOS-terminated, no truncation artifacts |
| T7 | Tool-call JSON | PASS | Output parsed as valid JSON with expected name/arguments |
| T8 | Vision smoke, 2 images | PASS | Identified the “HumanEval and HumanEval+ / Coding Quality” comparison table and the “BenchLocal / HermesAgent-20” horizontal grouped bar chart |
| T9 | Long answer (512 tokens) | PASS | 227 words; max repeated 15-gram = 1 |
| T10 | MTP throughput | PASS | 22.01 vs 20.22 tok/s mean (+8.9%); 21.92 vs 18.28 median (+19.9%) |
| T11 | Perplexity vs source BF16 | PASS | 1.8617 vs 1.8579 (+0.202%) |
| T12 | Next-token KL vs source BF16 | PASS | 0.01579 nats mean |
Isolated omlx serve on port 8123, --no-cache, thinking disabled, temperature=0, max_tokens=256, 3 prompts × 3 measured runs per configuration (warmup discarded):
| Configuration | Mean tok/s | Median tok/s | Mean completion tokens |
|---|---|---|---|
| MTP off | 20.22 | 18.28 | 138.0 |
| MTP on | 22.01 | 21.92 | 137.4 |
Across 22 MTP-on decode cycles observed during the benchmark (both variants), the MTP runtime reported mean draft acceptance 74.7% and 1.80 tokens/cycle. Throughput variance is high; the ship gate was ≥1.05× mean, which both variants clear.
Five held-out code snippets, 422 predicted tokens, compared against the unquantized source loaded sequentially:
| Metric | Source BF16 | This build |
|---|---|---|
| Perplexity | 1.8579 | 1.8617 |
| PPL delta | — | +0.202% |
| Mean next-token KL | — | 0.01579 nats |
oMLX discovers models as subdirectories of --model-dir:
omlx serve --model-dir /path/to/models --port 8000
The model id is the directory name: Kestrel-4B-Coder-oQ4e-mtp.
Enable MTP in ~/.omlx/model_settings.json:
{
"version": 1,
"models": {
"Kestrel-4B-Coder-oQ4e-mtp": {
"mtp_enabled": true,
"mtp_adaptive_max_depth": 2,
"mtp_fixed_depth": null,
"enable_thinking": false,
"temperature": 0.2,
"top_p": 0.95,
"top_k": 20
}
}
}
Stock
mlx_vlmsilently stripsmtp.*on load, andmlx_lmcannot load the vision tower. MTP plus vision requires oMLX ≥ 0.7.1 with its Qwen3.5 VLM runtime patch.
temperature=0.2, top_p=0.95, top_k=20temperature=1.0, top_p=0.95, top_k=20This build is entirely derivative. Credit for the foundational work:
Kestrel-4B-Coder: the source weights, model card, evaluations, and sampling guidance. Thank you for the foundational work this conversion is built on.unsloth_fixed_mtp fix).transformers, huggingface_hub, and safetensors.Not affiliated with or endorsed by any of the above. All trademarks belong to their respective owners.
Apache-2.0, inherited from the source model. See the included LICENSE. Upstream notices and attributions apply to this derivative.