d3-nano mixed-bit MLX

Unofficial MLX post-training quantization of vllm-sr/d3-nano (bf16, Apache-2.0). The source is a 24-layer Qwen3.5 text stack plus a vision tower and a 255-way decision readout. This package is not an official vLLM Semantic Router release.

Suggested repository id: Jakevin/d3-nano-ternary-mlx.

Method

The original bfloat16 checkpoint is the reference. For each language-model projection (attention Q/K/V/O, DeltaNet in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj, and MLP gate/up/down) one candidate replaces that projection alone and the KL of the decision readout's option distribution is measured. Candidates are ternary (one scale per group of 64, values -s, 0, +s), and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored in MLX's built-in 2-bit affine tensors so mlx-lm / mlx-vlm quantized matmul can run it. No custom Metal kernel is included.

Layer kinds are read from text_config.layer_types (24 layers: 18 DeltaNet, 6 full attention). Full attention is non-causal (attention_mode is noncausal_full_attention in the source package). DeltaNet stays causal.

Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the average storage width of the language projections between 3.40 and 3.50 bits. Ternary counts as 2 storage bits because that is the container it uses; it is selected only when its KL is lower than affine 2-bit. That additive minimum sums to 2.5403. Later swaps each raise one projection by one bit and lower another by one bit. A swap is kept only when the joint KL of the fully quantized model falls on 20 calibration questions and the average stays inside 3.40–3.50 bits. Joint KL on those questions went from 0.3599 to 0.1539 across 12 swaps. The shipped assignment's one-at-a-time sum is 2.9780. The decision readout is a separate matrix, last_hidden @ readout.T. It does not use the tied lm_head, so the token embedding is input-only and is stored at 8-bit affine (group 64). The decision readout, RMSNorms, DeltaNet convolution / A_log / dt_bias, and the entire vision tower stay at the original precision. The vision tower is not compressed.

Calibration: 64 questions from the author's decision-v7 development split (seed 20261011, one question from each drawn record) plus the three text questions in the source model card. 67 questions total. hard-v1 was not used.

Size

Language-projection storage mix (ternary counted as 2 bits). Average 3.4985 bits over 1,372,717,056 projection parameters. The tied embedding is 8-bit and is not in that average. The vision tower inside model.safetensors is uncompressed bf16.

Check against the source bf16 model

Same 150 questions as the d3-edge gate: 50 holdout questions, 50 decision-v7 questions and 50 transfer-v9 questions (seed 20261013, disjoint from these 67 calibration questions). Holdout and decision-v7 share 49 ids, so the merged 150 counts those twice, matching the list that was specified. One run, no confidence interval. Agreement is argmax against the source bf16 model loaded through this same MLX path. Accuracy is argmax against the question's label. KL is KL(bf16 || quantized).

Note: 141/150 was measured with an earlier mlx version. A re-run with mlx 0.32.2 gives 140/150 agreement (109/150 correct in both runs).

Vision weights were not quantized and were not part of this check.

Scores quoted from the source model card

The source card for vllm-sr/d3-nano reports, on its own evaluation and not on this package:

• Jev Decision Index 0.3, public suite: 42.22 (highlights; official 0.3 kit, 140,178 public requests).
• Jev Decision Index 0.3.1 table: index 37.2, public 42.2, same-skill tests 37.6, new-domain tasks 31.8.
• Jev Decision Index vision board 0.3.1: 58.1 (public 60.4, private 55.8).
• Perception Test, all 19,140 validation questions: 63.5.

Those numbers are copied from that card. They were not remeasured here.

Use

from pathlib import Path
from transformers import AutoTokenizer
import decide
import d3_mlx

root = Path(".")
model, config, decision, readout = d3_mlx.load_model(root)
tokenizer = AutoTokenizer.from_pretrained(root)
answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions)

From a shell, in this directory, python decide.py runs the three text questions from the source model card. python decide.py request.json reads {"state": ..., "questions": ...}.

mlx_lm can load the language tower because config.json sets model_file to d3_mlx.py, which builds layers from layer_types and uses stock MLX quantized linears. Stock Qwen3.5 code that assumes a fixed full-attention interval will not match this checkpoint. The vision weights use mlx-vlm's vision_tower.* names and stay bf16. Text decisions use decide.py.

License

Apache-2.0. Source: vllm-sr/d3-nano, itself built on Qwen/Qwen3.5-2B. See LICENSE and NOTICE.

繁體中文摘要

這是 vllm-sr/d3-nano 的非官方 MLX 量化包。 語言層每個投影各自試過三值、2、3、4 bit，先用一次換一層的 KL 在平均約 3.50 bit 的預算內分配， 再做 12 次對調（升高一層、降低另一層），只留下能降低 20 題聯合 KL 的對調。 三值借 MLX 內建 2-bit affine 儲存。決策頭是獨立 readout，不經過 tied 的輸出頭，嵌入用 8 bit。 決策頭、norm 與視覺塔維持原精度，視覺塔沒有壓縮。 model.safetensors 為 1890.5 MB，整個資料夾 1915.6 MB。 150 題（holdout 50 + decision-v7 50 + transfer-v9 50，單次、無信賴區間）與 bf16 答案一致 141/150，壓縮版正確 109/150，bf16 正確 111/150，平均 KL 0.0484，中位數 KL 0.0070。 來源模型卡的 Jev Decision Index 公開套件 42.22、視覺指數 58.1、Perception Test 63.5 是原文數字，這裡沒有重測。

Related release

A smaller ~3-bit build (131/150 agreement, below our 90% bar, published as-is) is at Jakevin/d3-nano-ternary-3bit-mlx.