primitive-ai/Kolibri-1-mixed-NVFP4-FP8

A 78 B model on one 96 GB card, 21% faster under load than the FP8 release.

Mixed NVFP4/FP8 build of Aleph-Alpha/Kolibri-1-BF16 at 45.81 GB, on one 96 GB card.
All 50 layers of routed experts at 4 bits; attention and shared experts are the FP8 weights Aleph Alpha ships; embeddings, lm_head, routers and norms stay BF16.

Two builds, one measurement. We quantized the experts two ways and ran both against Aleph Alpha's own FP8 release under one protocol. This build keeps the release's FP8 attention and is the faster. The NVFP4 build keeps attention in BF16 and is a little more accurate on knowledge. On tool calling all three are one band.

---

Why this quant

• 🗜️ 45.81 GB against 78.83 GB for the FP8 release. The release takes 73.55 GiB of a 96 GB card for weights; this build takes 42.78 GiB. The KV pool at 32K context grows from 410,315 tokens to 1,461,977, 3.6× as many.
• ⚡ 1,195 tok/s at concurrency 32, against 986 for the release (+21%). One user at a time it goes the other way: 142.6 tok/s against 159.1.
• 🎯 Tool calling is level with the release. 77.9 over five runs against 78.5 over five; the run-to-run spread is about 1.1 on both sides.
• 📉 Knowledge is about a point lower. 85.0 against 86.1. Both of this build's runs sit below both of the release's, and a gap of 1.1 is at the edge of what a 1,170-item suite resolves, so we report it as measured and do not call it a tie.
• 🧩 Only the experts changed. Attention and shared-expert weights are Aleph Alpha's FP8 bytes, copied unmodified, so none of the difference above comes from re-quantizing them.

---

Serve it

hf download primitive-ai/Kolibri-1-mixed-NVFP4-FP8 --local-dir ./Kolibri-1-mixed-NVFP4-FP8

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
  ghcr.io/aleph-alpha/aleph-alpha-inference:1.0.0-vllm0.29.0 \
  /models/Kolibri-1-mixed-NVFP4-FP8 --served-model-name kolibri \
  --kv-cache-dtype fp8 --max-model-len 32768 --gpu-memory-utilization 0.92 \
  --reasoning-parser kolibri1 --tool-call-parser kolibri1 --enable-auto-tool-choice

Stock vLLM does not know the Kolibri1ForCausalLM architecture. Aleph Alpha's plugin image adds it along with the kolibri1 parsers, and nothing here is patched on top of it. The image is vLLM 0.29.0 with aleph-alpha-inference 1.0.0 (sha256:9a56ab1691f8bc8eb0fc82f0f534bfdd36da2218fa79b01b83448739fbda403b). Boot takes about 130 s on an idle card, and the NVFP4 experts run on FlashInfer CUTLASS.

Aleph Alpha recommends temperature 1.0, top_p 0.97, top_k 128. Our numbers use the fixed protocol above for every build, so they are comparable with each other, not with Aleph Alpha's own. The measurement servers ran without the two parsers, because the tool-calling suite does not pass tools=. Thinking is on by default; send enable_thinking: false to turn it off.

---

Measured

One RTX PRO 6000 Blackwell, 96 GB, one card. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 32, auto-scored with no LLM judge. FP8 KV cache for every row, which is how Aleph Alpha evaluates the release. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell. Every row ran on the same box on the same day.

Tool-calling is the mean of five runs per build, with a within-build standard deviation of 0.6 to 1.2, so that column is one band. Knowledge is two runs per build (86.0/86.2, 85.9/86.5, 84.8/85.1 strict, in table order). On knowledge the release and NVFP4 are 0.1 apart; the mixed build is 1.1 below the release and 1.2 below NVFP4, on both of its runs. KV pool is the number vLLM reports at --max-model-len 32768, --gpu-memory-utilization 0.92.

What this table does not cover:

• The BF16 original. It is 156.2 GB and does not fit on one card, so Aleph Alpha's FP8 release is the reference row.
• German. Both suites are English. Kolibri is built for German and English, and we did not measure German.
• Context past 32K, and anything but Blackwell. The release is validated to 1M tokens; we did not test the 4-bit experts there. NVFP4 runs natively on Blackwell only.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above, the same 1,370 items, auto-scored, no LLM judge.

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with finished: overall scores an answer that overran the token budget as wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

---

What's quantized to what

Kolibri-1 is 78.10 B parameters, 3.46 B active per token, and 96.7% of them are routed experts.

compressed-tensors, two config groups, each with its own format: nvfp4-pack-quantized for the experts and float-quantized for attention and shared experts, with dynamic per-group FP8 activations. The release calls its block scale weight_scale_inv and compressed-tensors calls it weight_scale; the values are the same. vLLM only treats a group as activation-quantized when the group names one of its formats, so the top-level mixed-precision label alone fails at load with No compressed-tensors compatible scheme was found.

gate and up of every expert share one global scale, so the fused projection vLLM builds keeps the exact values. All 58,353 tensors of the BF16 source are accounted for, and loading logged no scale warning on any layer. Expert weights are round-to-nearest from the BF16 release, with no calibration.

---

_{primitive · more models · inference economics for production LLM systems}