⚡ Each donation = another big MoE quantized

I host 30+ free APEX MoE quantizations as independent research. My only local hardware is an NVIDIA DGX Spark (122 GB unified memory), enough for ~30-50B-class MoEs, but bigger ones (200B+) require rented compute on H100/H200/Blackwell, typically $20-100 per quant.
If APEX quants are useful to you, your support directly funds those bigger runs.

🎉 Patreon (Monthly)  |  ☕ Buy Me a Coffee  |  ⭐ GitHub Sponsors

Kolibri-1 APEX GGUF

APEX (Adaptive Precision for EXpert Models) quantizations of Aleph-Alpha/Kolibri-1.

Brought to you by the LocalAI team | APEX Project

A 78 B parameter MoE model, quantized to nine tiers.

Status: preview. Results below are a first pass with small sample sizes.

Benchmark Results

Measured with llama.cpp at commit 62318cd on an NVIDIA GB10. Perplexity and KL divergence are on wikitext-2. The reference is a Q8_0 conversion (83 GB), not BF16, because the 156 GB BF16 file does not fit in the 128 GB of memory used for the test. KL divergence and top-p agreement are relative to that Q8_0 file, which has a perplexity of 24.60 on this text. That is high for a 78 B model, so use the perplexity column to compare tiers with each other, not with other models. HellaSwag, Winogrande, ARC, MMLU and TruthfulQA use the llama.cpp multiple-choice mode with 400 tasks each (about ±2 to 2.5 points of noise). Speed is pp512 / tg128 in tokens per second.

Notes:

• KL divergence drops steadily from I-Micro to Quality and flattens at 0.081 to 0.084 for Quality, I-Quality, Balanced and I-Balanced. Balanced is no closer to the reference than Quality at 6.6 GB more, so Quality is the better pick.
• The quality benchmarks do not separate the mid and high tiers; the differences are inside the noise. The KL divergence and top-p columns are the reliable signal.
• Token generation is much faster on the small tiers (about 85 t/s at I-Micro against 59 t/s at Balanced).
• The I-Micro prompt-processing figure (992 t/s) comes from a single run and is probably a slow first run.

Files

I- files use an importance matrix built from diverse calibration data (chat, code, reasoning, tool-calling, agentic traces, Wikipedia). Quality, Balanced and Compact also ship without it.

What is APEX?

APEX is a quantization strategy for Mixture-of-Experts models. It classifies each tensor by role (routed expert, shared expert, attention) and applies a layer-wise precision gradient: the first and last layers keep higher precision, middle layers compress harder, and the always-active shared expert stays high.

The reason: in an MoE model the routed experts are most of the weights, but only a few of them fire for any given token. They tolerate lower precision than the parts every token passes through. See the APEX project for the recipes and scripts.

The model

Kolibri-1 is a Mixture-of-Experts model with 78 B parameters and about 3.5 B active per token. The source weights are FP8; these files were built from a BF16 conversion.

Usage

Kolibri-1 support is in llama.cpp master from commit 62318cd (PR 30673) or newer. Older builds cannot load these files.

llama-cli -m Kolibri-1-APEX-I-Mini.gguf -p "Your prompt" -ngl 99

Run with LocalAI

local-ai run mudler/Kolibri-1-APEX-GGUF@Kolibri-1-APEX-I-Mini.gguf

This needs a LocalAI llama.cpp backend built from a llama.cpp that includes the commit above.

License

These files are derivatives of Kolibri-1 by Aleph Alpha and fall under the base model license (Apache-2.0). Keep the license and attribution when you redistribute them.

Credits

APEX is brought to you by the LocalAI team. Built on llama.cpp.