Kolibri-1 — NVFP4 experts

Aleph Alpha's Kolibri-1 (78 B total / 3.46 B active MoE, German + English, reasoning, tool calling) with its routed experts re-quantized from FP8 to NVFP4. Everything else is the original FP8 checkpoint, bit for bit. 42.6 GiB instead of 73.4 GiB.

Built for, and measured on, a single NVIDIA DGX Spark (GB10): serving recipe at github.com/15ky3/Kolibri-1-DGX-Spark.

What changed

• The FP8 experts were dequantized exactly (to fp32) and quantized to NVFP4 with round-to-nearest from the per-group max, the same weight recipe ModelOpt uses. gate_proj and up_proj of each expert share one global scale, since vLLM fuses them. No calibration data was needed: activations stay BF16.
• The FP8 layers were not touched; their weight_scale_inv tensors are only renamed to weight_scale (same meaning) for the compressed-tensors format.
• Format: compressed-tensors (mixed: an FP8-block group and an NVFP4A16 group), written with compressed-tensors 0.17.0. Conversion script: tools/quantize_experts_nvfp4.py.
• Mean relative weight reconstruction error of the experts: 9.4 %.

Quality

Perplexity of reference texts scored inside the chat template (NVFP4 values are means of 3 runs; the Marlin MoE kernel is not bit-deterministic):

Needle-in-a-haystack passes from 21k to 978k tokens (1M context); reasoning split and tool calling work as in the original. German loses slightly more than English.

Speed on one DGX Spark (GB10)

vLLM 0.30.0, FP8 KV cache, decode of 600-token completions:

Prefill is unchanged (~5.5k tok/s at 21k tokens). The freed ~30 GiB go to the KV cache or to the host. On GB10 vLLM runs the experts with the Marlin NVFP4 MoE kernel (FP4 weights, BF16 math); other GPUs were not tested.

Serving

Kolibri needs Aleph Alpha's vLLM plugin aleph-alpha-inference. Its wheel pins vllm<0.30, but the code it uses is unchanged in vLLM 0.30.0, so it can be installed without dependencies:

pip install --no-deps 'aleph-alpha-inference==1.0.0'   # into a vLLM 0.30.0 environment
VLLM_USE_DEEP_GEMM=0 vllm serve iSkye/Kolibri-1-NVFP4-Experts \
  --kv-cache-dtype fp8 \
  --reasoning-parser kolibri1 --tool-call-parser kolibri1 --enable-auto-tool-choice

On a DGX Spark, the recipe does all of this with the stock vllm/vllm-openai:v0.30.0 image (plugin mounted, memory budget derived, watchdog), and serves this checkpoint by default.

Recommended sampling (from the original): temperature 1.0, top_p 0.97, top_k 128. Reasoning effort per request via chat_template_kwargs: {"reasoning_effort": "none" | "low" | "medium" | "high"}.

License and credits

Apache-2.0, like the original. This is a modified version of Aleph-Alpha/Kolibri-1 (© Aleph Alpha GmbH): the routed-expert weights were re-quantized as described above; nothing else was changed. All credit for the model goes to Aleph Alpha; please read the original model card for intended use, evaluation, risks and limitations — they apply here too.

Quantized with compressed-tensors (vLLM project); served with vLLM.