d3-lite mixed-bit MLX

Unofficial MLX post-training quantization of vllm-sr/d3-lite (bf16, Apache-2.0). The source is a 24-layer Qwen3.5 text stack plus a vision tower and a 255-way decision readout. This package is not an official vLLM Semantic Router release.

Suggested repository id: Jakevin/d3-lite-ternary-mlx.

Method

The original bfloat16 checkpoint is the reference. For each language-model projection (attention Q/K/V/O, DeltaNet in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj, and MLP gate/up/down) one candidate replaces that projection alone and the KL of the decision readout's option distribution is measured. Candidates are ternary (one scale per group of 64, values -s, 0, +s), and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored in MLX's built-in 2-bit affine tensors so mlx-lm / mlx-vlm quantized matmul can run it. No custom Metal kernel is included.

Layer kinds are read from text_config.layer_types: 18 DeltaNet layers and 6 full-attention layers. Full attention is non-causal, as in the source checkpoint. DeltaNet stays causal.

Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the average storage width of the language projections about 3.5 bits. Ternary counts as 2 storage bits because that is the container it uses; it is selected only when its KL is lower than affine 2-bit. That additive minimum sums to 3.7363. No joint bit-swap refinement was applied to this build. The shipped assignment's one-at-a-time sum is 3.7363. The decision readout is a separate matrix, last_hidden @ readout.T. It does not use the tied lm_head, so the token embedding is input-only and stays 8-bit affine (group 64). The decision readout, RMSNorms, DeltaNet convolution / A_log / dt_bias, and the entire vision tower stay at the original precision.

Calibration: 64 questions from the author's decision-v7 development split (seed 20261011, one question from each drawn record) plus the three text questions in the source model card. 67 questions total. hard-v1 was not used.

Size

Language-projection storage mix (ternary counted as 2 bits). Average 3.500 bits over 497,614,848 projection parameters. The tied embedding is 8-bit and is not in that average. The vision tower inside model.safetensors is uncompressed bf16.

Evaluation

Single run, no confidence interval. 150 questions: 50 holdout from decision-v7 (seed 20261013, disjoint from calibration and the 8 smoke questions) plus 50 decision-v7 and 50 transfer-v9 (same seed, disjoint from calibration). Agreement is argmax vs the source bf16 model on the same MLX decision path. Accuracy is argmax vs the dataset label.

Overall agreement 130/150 (86.7%). Mean KL(bf16 || quantized) 0.0407, median 0.0097. Compressed accuracy 99/150, bf16 accuracy 98/150.

Gate note. This build does not reach 90% agreement with bf16. For models under 1B the release gate is: compressed accuracy not below bf16 on these 150 questions and mean KL <= 0.05. This build has 99 vs 98 correct and mean KL 0.0407.

The 20 disagreements are mostly questions where bf16 itself was unsure: bf16 top-1 probability median 0.50 (top-1 minus top-2 margin median 0.10) vs 0.96 (margin 0.92) on the 130 agreeing questions. Of the 20, 7 went correct->wrong, 8 wrong->correct and 5 wrong->wrong.

The 150 questions are not fully independent: the holdout and decision-v7 sets share 49 of their 50 questions.

Reload check: eight other questions scored from the in-memory quantized model and again after loading out/release/ from disk. Agreement with memory 8/8.

The vision tower was not quantized and is untested in this package; it was not part of this evaluation.

Official source-card scores, quoted from vllm-sr/d3-lite (Jev Decision Index 0.3.1, source evaluation, not reproduced here): Decision Index 29.0, public suite 36.9, same-skill tests 31.6, new-domain tasks 21.6. Vision Index 46.9. Perception Test (internal) 59.3.

Use

from pathlib import Path
from transformers import AutoTokenizer
import decide
import d3_mlx

root = Path(".")
model, config, decision, readout = d3_mlx.load_model(root)
tokenizer = AutoTokenizer.from_pretrained(root)
answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions)

From a shell, in this directory:

python decide.py

That runs the three text questions from the source model card. python decide.py request.json reads {"state": ..., "questions": ...}.

mlx_lm can load the language tower because config.json sets model_file to d3_mlx.py, which builds layers from layer_types and uses stock MLX quantized linears. The vision weights use mlx-vlm's vision_tower.* names and stay bf16. Text decisions use decide.py.

License

Apache-2.0. Source: vllm-sr/d3-lite, itself built on Qwen/Qwen3.5-0.8B. See LICENSE and NOTICE.

繁體中文摘要

這是 vllm-sr/d3-lite 的非官方 MLX 量化包。 語言層的每個投影各自試過三值、2、3、4 bit，先用一次換一層的 KL 在平均約 3.50 bit 的預算內分配， 未做聯合對調。三值借 MLX 內建 2-bit affine 儲存。決策頭是獨立 readout，不經過 tied 的輸出頭， 所以嵌入維持 8 bit。決策頭、norm 與視覺塔維持原精度，視覺部分未壓縮。 model.safetensors 為 721.3 MB，整個資料夾 745.4 MB。 150 題（單次、無信賴區間）與 bf16 一致 130/150， 壓縮版正確 99/150，平均 KL 0.0407。 與 bf16 的不一致多半發生在 bf16 本身沒把握的題目（20 題中 7 題對→錯、8 題錯→對、5 題錯→錯）。holdout 與 decision-v7 有 49/50 題相同。視覺部分未壓縮、未測試。