d3-edge-W4A16-HGPTQ
4-bit Walsh–Hadamard + Cholesky GPTQ (W4A16-HGPTQ, group size 64) quantization of vllm-sr/d3-edge (vllm-sr/d3 code-readout-v1 single-pass decision model). It compresses the 1.11 GiB BF16 checkpoint (readout.safetensors + backbone) to 0.65 GiB (1.71× smaller) while preserving 100.00% top-1 decision agreement and 0.995193 readout logit cosine similarity against the BF16 reference.
- Footprint:
0.65 GiBtotal across78quantized text linears, plus50vision tower & merger linears, plus the exact BF16readout.safetensors(255 × 1024code-readout head). - Quantization recipe: 64×64 Sylvester Walsh–Hadamard input rotation (
H_64) + MSE-initialized Cholesky GPTQ (group_size=64,actorder="static",dampening_frac=0.01), calibrated layer-by-layer undernoncausal_full_attentionattention. - Architecture & recurrence:
12layers (hidden_size=1024). Single-pass hybrid GatedDeltaNet + full-attention forward pass. - Canonical weight layout: All safetensors keys use canonical
model.language_model.*(andmodel.visual.*) prefixes with"quantization_status": "compressed", loading cleanly withmissing: 0, unexpected: 0inQwen3_5Model.from_pretrained,D3.from_pretrained, andwarp-34.
Quality vs. BF16 Base Model (vllm-sr/d3-edge)
Single-pass code-readout-v1 evaluation across multi-question choice, yes_no, multi_select, and scale decision tasks (with forward + reverse option-permutation averaging):
| Evaluation | BF16 Base (vllm-sr/d3-edge) |
d3-edge-W4A16-HGPTQ |
Notes |
|---|---|---|---|
Safetensors Size (incl. readout.safetensors) |
1.11 GiB (1,189,722,528 B) |
0.65 GiB (695,386,416 B) |
1.71× compression (−0.46 GiB) |
Decision Top-1 Agreement (12 questions) ↑ |
100.00% |
100.00% |
Forward + reverse permutation-averaged readout |
| Option Probability KL Divergence (mean / max nats) ↓ | 0.000000 |
0.000934 / 0.005042 |
Per-question normalized option distribution KL |
| 255-Code Readout Logit Cosine Similarity ↑ | 1.000000 |
0.995193 |
Full 255-code softmax KL 0.008751 nats |
Vision Tower (vision-probe-v2) (96 images, 50,596 visual tokens) |
1.000000 cos sim |
0.974388 mean cos (0.953108 min) |
vision-guard-v2 mean cos 0.974515 (min 0.953134) |
What Is Quantized
- Quantized (
128linear layers inW4A16-HGPTQ):- Language (
78linears): All12hybrid GatedDeltaNet + full-attention blocks (q_proj,k_proj,v_proj,o_proj,in_proj_qkv,in_proj_z,out_proj,gate_proj,up_proj,down_proj) with online block-Hadamard rotation (head_dim=64) + Cholesky GPTQ (group_size=64,actorder="static"). - Vision (
50linears): All 12 ViT blocks (visual.blocks.*.{attn.qkv,attn.proj,mlp.linear_fc1,mlp.linear_fc2}) plusvisual.merger.{linear_fc1,linear_fc2}ing64-h64.
- Language (
- Kept in BF16:
readout.safetensors(weight[255, 1024]),embed_tokens,linear_attn.in_proj_a,linear_attn.in_proj_b, norms, biases, andvisual.patch_embed.proj.
Usage
from d3_runtime import D3
model = D3.from_pretrained("nchapman/d3-edge-W4A16-HGPTQ", device="cuda:0")
result = model.system_one(
state="Customer ordered item #4412, delivered on time, requested an invoice copy.",
questions={
"intent": {
"type": "choice",
"instructions": "What does the customer need?",
"criteria": {"invoice": "Copy of invoice", "refund": "Return and refund"},
}
},
)
print(result)