eins78/Kolibri-1-mlx-5bit

Aleph-Alpha/Kolibri-1 converted to MLX with stock mlx-vlm 0.7.8 convert, from the BF16 release Aleph-Alpha/Kolibri-1-BF16. Quantisation: affine 5-bit, group size 64, for all linear layers except the router; embedding and lm_head in bf16. 5.59 bits per weight on average, 54.6 GB on disk. It is 54.6 GB on disk; a 64 GB Mac with nothing else running may be enough, which was not tested; 96 GB is the safe choice.

Kolibri 1 is Aleph Alpha's German/English mixture-of-experts model (78B parameters, 50 layers, 384 experts, 6 routed plus 1 shared expert per token, sliding-window attention of 513 tokens in a 4:1 sliding:full pattern). Refer to the original model card for what the model is for.

This is one rung of a set; all rungs were measured the same way (table below).

Use with mlx-vlm

No fork and no extra code: mlx-vlm 0.7.8 and later include the kolibri1 architecture.

pip install -U "mlx-vlm>=0.7.8"
python -m mlx_vlm generate --model eins78/Kolibri-1-mlx-5bit --max-tokens 400 \
  --temperature 1.0 --top-p 0.97 --top-k 128 \
  --prompt "Erkläre in zwei Sätzen, warum der Himmel blau ist."

from mlx_vlm import load, generate

model, processor = load("eins78/Kolibri-1-mlx-5bit")
tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor
messages = [{"role": "user", "content": "Erkläre in zwei Sätzen, warum der Himmel blau ist."}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False, reasoning_effort="low"
)
result = generate(model, processor, prompt, max_tokens=600, temperature=1.0, top_p=0.97, top_k=128)
print(result.text)

Recommended sampling (Aleph Alpha): temperature 1.0, top_p 0.97, top_k 128. The chat template switches reasoning with reasoning_effort ("none", "low", "medium", "high") or enable_thinking=False; with reasoning on the model writes <think>...</think> first. Tool calls come out as Hermes-style <tool_call>{"name": ..., "arguments": ...}</tool_call> (pass tools=[...] to apply_chat_template).

Not for mlx-lm: the weight names follow mlx-vlm's layout (language_model. prefix), so mlx_lm.load does not read these files, even with the open mlx-lm pull request for Kolibri. For mlx-lm, see eins78/Kolibri-1-mlx-mixed-4-8-bit.

The set

Top-1 token agreement / mean KL divergence against the unquantised model in bf16, on two prompts longer than the 513-token window (English 618 tokens, German 612 tokens), scored at every position of the prompt. Higher top-1 and lower KL are better. For scale, bf16 itself against fp32 scores 0.977 / 0.0066 (en_long) and 0.923 / 0.0318 (de_long).

There is no bf16 MLX copy: stock mlx-vlm loads Aleph-Alpha/Kolibri-1-BF16 directly.

Measured quality of this model

Baselines: bf16 is the unquantised BF16 release run in bf16 through mlx-vlm's own layers; fp32 is the same weights run in fp32. The last column shows how far bf16 itself is from fp32.

The short prompts (13 to 24 tokens) are noisy even for bf16: the first positions of a context carry very large activations, and small errors there change the top-1 token more often.

Not verified here

This model is 54.6 GB. On the 64 GB Mac used for this release, with other services running, loading it went into swap, so loading and generation were not tested. The quality numbers above are measured on these exact files: the weights were read one decoder layer at a time and the prompt was run through all 50 layers (with causal attention this gives the same logits as a full-model prefill; for the rungs that fit, the two agree to the digit). Speed and memory were not measured. The weights load with mlx_vlm.load the same way as the tested rungs, and they were written by the same convert call with a different --q-bits.

How this was verified

1. The mlx-vlm model code is exact. mlx-vlm 0.7.8's own kolibri1 layers, run in fp32 one layer at a time from the embedding to the logits on the FP8 checkpoint, reproduce a fp32 PyTorch reference written from Aleph Alpha's vLLM plugin: top-1 1.000 and top-5 1.000 on five German and English prompts, mean KL about 1e-11, identical expert choice at every layer. So every difference measured on this card is precision (bf16 and quantisation), not a wrong implementation. The reference is a torch rebuild of the plugin's math; the plugin itself needs CUDA and was not run.
2. Each rung is measured on its own files against bf16 and fp32 runs of the BF16 weights, through the same mlx-vlm layers.
3. Rungs that fit in 64 GB were also loaded whole with mlx_vlm.load and given the smoke and speed test.

Scripts and method: kolibri-mlx, section 11. Stock mlx-vlm and the MLX release (scripts/vlm_chain.py, scripts/vlm_e2e.py, scripts/vlm_smoke.py, scripts/vlm_convert.py).

AI assistance

The conversion, the verification scripts, the measurements and this model card were produced by an AI coding agent (Claude Code, Claude Opus 5.5) working from a written brief, directed and reviewed by the repository owner (eins78). The numbers on this card were generated from the result files by script, not typed by hand. The upload was done by the agent with the owner's approval.

License

Apache License 2.0, inherited from Aleph Alpha's Kolibri-1-BF16 (the LICENSE file is included). Model weights and tokenizer are by Aleph Alpha; this repository only changes the format and precision. Aleph Alpha publishes no acceptable use policy for Kolibri and asks users to use it lawfully; see the Responsible Use section of the original card.