d3-oQ4e
[!NOTE] Updated 2026-11-19: Re-quantized with the refined oQe rounding in oMLX 0.8.0, which lowers quantization error at the same size and speed. Please re-download if you have an earlier copy.
This model was quantized using oQ (oMLX v0.8.0) mixed-precision quantization with improved oQe (jundot/omlx#4930) and oQe rounding refinement (jundot/omlx#5292).
A mixed-precision MLX checkpoint of vllm-sr/d3, the 27B multimodal decision model of Decision 3.0 from vLLM Semantic Router, built on Qwen/Qwen3.8-27B. It reads text or JSON with optional images and videos and returns a probability for every option of choice, yes/no and score questions without generating text. It includes the language backbone, the vision encoder and the code readout.
The code readout (readout.safetensors, FP32) and decision_config.json are copied unchanged from the source. oMLX serves this checkpoint through POST /v1/systemone; Decision 3.0 support (jundot/omlx#4470) is newer than oMLX v0.7.1.dev1.
The effective average is 4.829 bits/weight for the language backbone and 5.027 bits/weight for the complete checkpoint.
Quantization and Bit Distribution
oQ4e allocates bits by measured layer sensitivity, then rounds each group with an importance matrix (128 samples x 512 tokens) and a weighted least-squares refit of its scale and bias.
| Storage format | Logical weights | Share of total | Tensor storage | Effective bits/weight |
|---|---|---|---|---|
| Affine 4-bit, group 64 | 17.412B | 66.748% | 9.122 GiB | 4.50 |
| Affine 5-bit, group 64 | 8.031B | 30.787% | 5.142 GiB | 5.50 |
| Affine 6-bit, group 64 | 0.178B | 0.683% | 0.135 GiB | 6.50 |
| BF16 | 0.463B | 1.776% | 0.863 GiB | 16.00 |
| FP32 | 0.001B | 0.005% | 0.005 GiB | 32.00 |
| Total | 26.087B | 100% | 15.267 GiB | 5.027 |
Storage Breakdown
| Component | Logical weights | Storage |
|---|---|---|
| Language backbone | 25.625B | 15.466 GB / 14.404 GiB |
| Vision encoder | 0.461B | 0.921 GB / 0.858 GiB |
| Code readout | 0.001B | 0.005 GB / 0.005 GiB |
| Total | 26.087B | 16.393 GB / 15.267 GiB |
License
Apache-2.0, following the source model. See LICENSE and NOTICE.