Attention in FP8, most of the MLP in NVFP4: 40% less drift than plain NVFP4 for 3.23 GB more.
Mixed NVFP4 / FP8 build of Nanbeige/Nanbeige4.3-33B at 25.30 GB, against 66.24 GB for the BF16 release.
216 of the 364 decoder linear layers are FP8 W8A8; the rest are NVFP4 W4A4. Embeddings, the LM head and the norms stay BF16.
Drift is the extra negative log-likelihood per token a build assigns to 400 reasoning traces that BF16 wrote on the same card; top-1 agreement is how often its most likely next token is BF16's. The pass is deterministic.
Why this build
- 🎯 Drift +38.9×10⁻³ nats per token from BF16, against +64.3 for plain NVFP4 and +5.95 for FP8. Top-1 agreement 95.1%, against 93.7% and 97.9%.
- 🗜️ 25.30 GB: 3.23 GB more than NVFP4 (22.07 GB) and 10.23 GB less than FP8. KV pool 311,488 tokens at a 32,768-token window.
- 📏 Knowledge 88.9, tool calls 79.0, abstain 63.3; BF16 scores 89.7, 80.8 and 63.3. 99.1% of answers finish inside the budget.
- ⚡ 48.0 tok/s single-stream and 308 at concurrency 32 on one card, against 20.1 and 129 for BF16 on the same card.
Serve it
hf download primitive-ai/Nanbeige4.3-33B-mixed-NVFP4-FP8 --local-dir ./Nanbeige4.3-33B-mixed-NVFP4-FP8
docker run --gpus '"device=0"' --ipc=host -p 8000:8000 -v $PWD:/models \
vllm/vllm-openai:v0.32.0 \
--model /models/Nanbeige4.3-33B-mixed-NVFP4-FP8 \
--max-model-len 131072 --gpu-memory-utilization 0.92 \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
Checked with exactly this command on an idle card: boot 181 s, 23.90 GiB of weights, a 298,832-token KV pool at the 131,072-token window. What is 17*23? comes back as 391 with the reasoning split off by the parser, and as 391 with chat_template_kwargs: {"enable_thinking": false}; a weather question comes back as the parsed call get_weather {"city": "Lisbon"}. The model's window is 524,288 tokens; raise --max-model-len as far as the KV pool allows.
Measured
One RTX PRO 6000 Blackwell, 96 GB, no tensor parallelism, vLLM 0.32.0, BF16 KV cache. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell. KV pool is at a 32,768-token window and --gpu-memory-utilization 0.92.
| build | format | size | drift (×10⁻³ nats/token) | top-1 agreement | knowledge | call | abstain | finished | tok/s @1 | tok/s @32 | KV pool (tokens) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Nanbeige4.3-33B (BF16 release) | BF16 | 66.24 GB | 0 (reference) | 100% | 89.7 | 80.8 | 63.3 | 99.2% | 20.1 | 129 | 113,648 |
| FP8 | FP8 W8A8, every Linear | 35.53 GB | +5.95 | 97.9% | 89.3 | 80.4 | 63.3 | 99.1% | 35.4 | 225 | 262,448 |
| this repo | NVFP4 W4A4, FP8 W8A8 in 216 modules | 25.30 GB | +38.89 | 95.1% | 88.9 | 79.0 | 63.3 | 99.1% | 48.0 | 308 | 311,488 |
| NVFP4 | NVFP4 W4A4, every Linear | 22.07 GB | +64.28 | 93.7% | 89.0 | 78.8 | 64.2 | 99.1% | 53.8 | 336 | 326,144 |
Drift and agreement come from 400 BF16 traces of the knowledge suite (215,950 completion tokens); the standard error over traces is 1.1 for this build. Tool calling is 3 runs per build; knowledge is 2 runs per quantized build and 1 for BF16.
What this table does not cover:
- Context past 32K. The model's window is 524,288 tokens; we did not measure the quantized layers there.
- Other GPUs. Only the RTX PRO 6000 Blackwell (sm_120) was tested.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above, the
same 1,370 items, auto-scored, no LLM judge.
| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
| Nemotron-3.5-Lightning-30B-A3B | 30 B MoE+Mamba | 19.2 GiB | 87.1 | 87.9 | 85.4 | 70.5 | 97.9% | 1429 |
| Ornith-1.5-35B-A3B | 35 B MoE | 22.6 GiB | 88.7 | 91.7 | 74.4 | 60.0 | 99.3% | 760 |
| Muse-Glimmer-30B | 30 B MoE | 20.4 GiB | 86.6 | 88.8 | 78.6 | 54.5 | 99.7% | 800 |
| Qwen3.8-27B | 27 B dense | 20.7 GiB | 88.8 | 90.4 | 85.5 | 54.5 | 99.7% | 651 |
| Granite-4.2-30B | 30 B dense | 18.1 GB | 85.5 | 86.2 | 85.8 | 60.8 | 98.5% | 1502 |
| Nex-N2.5-mini NVFP4 | 35 B MoE, 3 B active | 23.91 GB | 88.5 | 90.5 | 81.9 | 57.5 | 99.3% | 524 |
| Nex-N2.5-mini mixed | 35 B MoE, 3 B active | 26.04 GB | 88.5 | 90.6 | 80.9 | 58.7 | 99.2% | 504 |
| Nex-N2.5-mini FP8 | 35 B MoE, 3 B active | 38.13 GB | 88.7 | 90.9 | 79.4 | 60.0 | 99.5% | 545 |
| K2-Horizon-MoVA-36B-A4B NVFP4 | 37 B MoE+MoVA, 4 B active | 36.7 GB | 84.1 | 86.5 | 71.8 | 60.4 | 95.6% | 1234 |
| K2-Horizon-MoVA-36B-A4B mixed | 37 B MoE+MoVA, 4 B active | 44.5 GB | 84.9 | 87.3 | 73.5 | 58.9 | 96.3% | 1118 |
| Nemotron-3.5-Bolt-74B-A9B mixed | 75 B MoE+Mamba, 9 B active | 46.70 GB | 89.3 | 90.6 | 87.9 | 58.3 | 98.5% | 1173 |
| Laguna-S-2.1 | 110 B MoE | 64.0 GiB | 84.3 | 87.1 | 64.6 | 81.0 | 97.3% | 995 |
| Qwen3.8-Flash-Next | 180 B MoE, 6 B active | 183.7 GB | 90.3 | 92.2 | 84.8 | 56.7 | 99.5% | 686 |
| MiMo-V2.6-Flash-RL NVFP4 | 309 B MoE, 15 B active | 182.40 GB | 89.3 | 91.1 | 83.0 | 61.0 | 99.0% | 664 |
| Kolibri-1 FP8 | 78 B MoE, 3.5 B active | 78.83 GB | 85.0 | 86.1 | 86.1 | 48.0 | 97.0% | 1824 |
| Kolibri-1 NVFP4 | 78 B MoE, 3.5 B active | 47.70 GB | 84.9 | 86.2 | 84.9 | 48.0 | 96.4% | 1947 |
| Kolibri-1 mixed | 78 B MoE, 3.5 B active | 45.81 GB | 83.9 | 85.0 | 85.5 | 47.5 | 96.6% | 1848 |
| LFM2.5-24B-A2B NVFP4A16 | 24 B MoE, 2.45 B active | 14.95 GB | 83.9 | 85.4 | 82.4 | 44.5 | 98.8% | 1603 |
| MiniMax-M3.1-Flash (release, 4 cards) | 427 B MoE, 23 B active | 250.12 GB | 82.9 | 85.0 | 81.7 | 25.0 | 99.2% | 513 |
| MiniMax-M3.1-Flash-pruned-272B | 272 B MoE, 23 B active | 162.90 GB | 82.5 | 84.4 | 80.4 | 32.5 | 98.8% | 707 |
| Nanbeige4.3-33B (BF16 release) | 33 B dense | 66.24 GB | 87.9 | 89.7 | 80.8 | 63.3 | 99.2% | 846 |
| Nanbeige4.3-33B FP8 | 33 B dense | 35.53 GB | 87.5 | 89.3 | 80.4 | 63.3 | 99.1% | 871 |
| Nanbeige4.3-33B mixed (this repo) | 33 B dense | 25.30 GB | 87.0 | 88.9 | 79.0 | 63.3 | 99.1% | 905 |
| Nanbeige4.3-33B NVFP4 | 33 B dense | 22.07 GB | 87.1 | 89.0 | 78.8 | 64.2 | 99.1% | 940 |
overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. Read it with finished: overall scores an answer that overran the token budget as
wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in
overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.
What's in it
Nanbeige4.3-33B is a dense Llama-architecture model: 33.12 B parameters, 52 layers, hidden size 7168, 56 query and 8 KV heads, MLP width 22,016.
| params | share | |
|---|---|---|
| MLP gate + up + down, 52 layers | 24.62 B | 74.3% |
| attention q + k + v, 52 layers | 3.44 B | 10.4% |
| attention o, 52 layers | 2.67 B | 8.1% |
| embeddings and LM head (untied) | 2.39 B | 7.2% |
| modules | count | format |
|---|---|---|
| attention q, k, v, o, layers 0-51 (all) | 208 | FP8 E4M3, one scale per output channel; activations FP8, one scale per token |
| MLP gate, up, layers 18, 19, 32, 33 | 8 | FP8 E4M3, one scale per output channel; activations FP8, one scale per token |
| every other decoder Linear | 148 | NVFP4 (E2M1, E4M3 scale per 16, FP32 global); activations NVFP4 per 16 |
| embeddings, LM head, norms | all | BF16, byte-identical to the release |
The checkpoint is compressed-tensors and runs on vLLM's stock kernels. q/k/v and gate/up share one format per layer because vLLM fuses them.
Licence
Apache-2.0, the base model's licence. This is a modified version of Nanbeige/Nanbeige4.3-33B: the decoder's linear layers were quantized and config.json carries a quantization_config. Everything else is unchanged.
primitive ·
more models ·
inference economics for production LLM systems