Qwen3.6-27B — PrismaQuant gridbook 5.5-bit (gridbook formats, vLLM)
23.0 GB codebook-quantized Qwen3.6-27B (vision-language, MTP included) serving through stock vLLM with the open gridbook plugin — no fork, no core patches. Weights are stored as product-VQ codebook indices whose codebook values lie exactly on hardware grids (e4m3 / E2M1), so serving decodes into bit-standard FP8/NVFP4 tensors and runs on the same tensor-core GEMM paths a conventional checkpoint uses.
Measured quality (held-out KL vs the BF16 teacher)
Protocol: top-20 prompt_logprobs KL on a 12.3k-token held-out corpus
(disjoint from calibration), 8,176 positions, all arms served back-to-back on
one box against the same teacher capture — a strict relative comparison.
| artifact | size | KL (all) | KL (confident half) | top-1 (conf) |
|---|---|---|---|---|
| this artifact (CB ladder, 5.5 bpp body) | 23.0 GB | 0.0049 | 0.00295 | 99.88% |
| Qwen3.6-27B-PrismaAURA-5.5bit (NVFP4+FP8) | 23 GB | 0.0211 | 0.0130 | 99.58% |
| Qwen3.6-27B-PrismaSCOUT 5p31 | 20.6 GB | 0.0344 | 0.0249 | 99.04% |
At matched size and matched body bitrate, the codebook ladder carries −77% KL versus the strongest conventional (NVFP4+FP8) artifact in this family. Same model, same bytes, same protocol — the delta is the format.
- Pre-ship validator: PASS — mean NLL 1.400/tok, p99 NLL 1.807 (thresholds 3.0 / 6.0)
- ToolEvalBench (hardmode, seed 1234, sequential): 87 / 100 (129/148) — family anchors on identical protocol: BF16 teacher 86, PrismaSCOUT 85, PrismaAURA-5.5 91; repeat churn band ±2–3
What's inside
| section | format | notes |
|---|---|---|
| 504 body Linears | FP8-CB K36–K47 (426 units) + BF16 (69) | per-Linear rungs chosen by measured error under a byte budget; K47 (5.88 bpw) ×175, K44 ×109, K40 ×108 — the ⅛-bpw all-integer ladder in actual use |
| vision tower (110 Linears) | NVFP4 weight-only (W4A16) | quantized ViT, verified on image inputs |
| MTP draft block | BF16 | canon throughput selector: highest-fidelity rung that fits |
| embeddings + lm_head | BF16 | 248k vocab, untied |
Full per-Linear assignment ships in allocation/. Codebooks (cb_codebooks.pqcb)
are shared per role family and total a few MB.
Serve this model
| Plugin | gridbook — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches. |
| GPU | NVIDIA Blackwell, compute capability sm_120 / sm_121. Measured on GB10 / DGX Spark (sm_121). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target. |
| Memory | 20.8 GiB weights + KV cache, measured on one 128 GB unified-memory GB10 (Model loading took 20.82 GiB). Before gridbook 0.1.0 the same artifact loaded at 35.86 GiB — dense codebook weights were resident twice (issue #1, fixed in 0.1.0); use 0.1.0 or newer. A 32 GB consumer Blackwell (RTX 5090, sm_120) should now fit weights plus a useful KV budget, and the issue reporter measured 20.54 GiB with 32k context using equivalent patches — but we have not verified 0.1.0 on a 32 GB card ourselves. |
| vLLM | Served here on the vllm-node image used for the 0.1.0 verification; also exercised on 0.25.1 and 0.23.1rc1.dev1060 by an external reporter (issue #1). The plugin surface is small but vLLM internals drift — pin a version you have tested. |
| Toolchain | CUDA toolkit with nvcc on PATH in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). nvcc 13.0 is the tested toolchain. |
| Parallelism | Single GPU (tp=1). Tensor parallelism is not implemented in the plugin. |
pip install gridbook # JIT-builds kernels for your GPU (capability >= 8.0 floor)
vllm serve rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm \
--max-model-len 32768 --gpu-memory-utilization 0.90
Honest limitations
- KL numbers are the relative protocol above, not the full-vocab exact-KL contract; the ranking is protocol-internal and same-session.
- ToolEvalBench ordering vs PrismaAURA-5.5 may differ from the KL ordering (task metrics churn ±2–3 points between serving configs).
- One box, one seed per benchmark.
Quantized with PrismaQuant (measurement-driven per-Linear format allocation); format spec and kernels: gridbook, Apache 2.0.
Contact: robert.tand@icloud.com