Kolibri-1 Heretic — NVFP4 experts
iSkye/Kolibri-1-heretic,
the decensored FP8 version of Aleph Alpha's Kolibri-1,
with its routed experts re-quantized from FP8 to NVFP4. Everything else is
the heretic FP8 checkpoint, bit for bit. 42.6 GiB instead of 74 GB.
[!WARNING] This model has had most of its refusal behavior removed. It will answer requests the original model declines. You are responsible for how you use it and for complying with applicable laws and the Apache-2.0 license.
Built for, and measured on, a single NVIDIA DGX Spark (GB10): serving
recipe at github.com/15ky3/Kolibri-1-DGX-Spark
(ABLIT=1).
Lineage
| step | checkpoint | what changed |
|---|---|---|
| 1 | Aleph-Alpha/Kolibri-1 | the original, FP8 128×128 blocks |
| 2 | iSkye/Kolibri-1-heretic | refusals removed with Heretic (ARA, LoRA rank 50 merged): self_attn.o_proj and mlp.shared_experts.down_proj only |
| 3 | this repo | routed experts FP8 → NVFP4 (weight-only, NVFP4A16) |
Because Heretic never touched the routed experts, the NVFP4 experts here are
bit-identical to those of
iSkye/Kolibri-1-NVFP4-Experts;
the two checkpoints differ only in o_proj and the shared experts' down_proj.
Format
| part | format |
|---|---|
| routed experts (384 × 50 layers) | NVFP4 weight-only: FP4 E2M1, group 16, FP8 E4M3 group scales, fp32 global scale (gate/up share it) |
| attention q/k/v/o, shared expert | FP8 128×128 blocks, as in the heretic checkpoint |
| router, norms, embeddings, LM head | unchanged |
Round-to-nearest from the per-group max, no calibration data (activations stay
BF16). Format compressed-tensors (mixed FP8-block + NVFP4A16 groups), written
with compressed-tensors 0.17.0 by
tools/quantize_experts_nvfp4.py.
Quality
Perplexity of reference texts inside the chat template (NVFP4 values: mean of 3 runs, the Marlin MoE kernel is not bit-deterministic):
| text | original FP8 | heretic FP8 | heretic NVFP4 |
|---|---|---|---|
| German, memorized | 1.675 | 2.327 | 2.398 |
| English, memorized | 1.126 | 1.221 | 1.258 |
| German, freshly written | 8.385 | 9.548 | 9.734 |
| English, freshly written | 13.78 | 13.97 | 14.31 |
NVFP4 adds about +2–3 % on top of the heretic FP8; the abliteration itself costs more, especially in German. Refusal behaviour follows the heretic checkpoint (3/100 refusals in Heretic's evaluation, original 100/100). Reasoning split and tool calling work as in the original.
Speed on one DGX Spark (GB10)
vLLM 0.30.0, experts on the Marlin NVFP4 MoE kernel, FP8 KV cache: 54 tok/s single stream, 272 tok/s aggregate at 8 streams (heretic FP8: ~48 / ~179). 42.8 GiB on the GPU instead of 73.6.
Serving
Needs Aleph Alpha's vLLM plugin
aleph-alpha-inference,
installed without dependencies into vLLM 0.30.0 (its pin says <0.30, the code
it uses is unchanged):
pip install --no-deps 'aleph-alpha-inference==1.0.0'
VLLM_USE_DEEP_GEMM=0 vllm serve iSkye/Kolibri-1-heretic-NVFP4-Experts \
--kv-cache-dtype fp8 \
--reasoning-parser kolibri1 --tool-call-parser kolibri1 --enable-auto-tool-choice
On a DGX Spark: ABLIT=1 ./download.sh && ABLIT=1 ./start.sh in the
recipe.
License and credits
Apache-2.0. A modified version of Aleph-Alpha/Kolibri-1 (© Aleph Alpha GmbH): abliterated with Heretic (iSkye/Kolibri-1-heretic), then routed experts re-quantized as described above. All credit for the model goes to Aleph Alpha; read the original model card for intended use, evaluation, risks and limitations. Quantized with compressed-tensors, served with vLLM.