eins78/Kolibri-1-mlx-mixed-4-8bit-vlm

Aleph-Alpha/Kolibri-1 converted to MLX with stock mlx-vlm 0.7.8 convert, from the BF16 release Aleph-Alpha/Kolibri-1-BF16. Quantisation: mixed affine, group size 64: routed experts (switch_mlp) 4-bit; attention, shared expert, embedding and lm_head 8-bit; router in bf16. 4.64 bits per weight on average, 45.3 GB on disk. Peak memory was 45.9 GB on a 64 GB Mac (618-token prompt, 256 new tokens), so it needs a Mac with 64 GB or more; long contexts need more.

Kolibri 1 is Aleph Alpha's German/English mixture-of-experts model (78B parameters, 50 layers, 384 experts, 6 routed plus 1 shared expert per token, sliding-window attention of 513 tokens in a 4:1 sliding:full pattern). Refer to the original model card for what the model is for.

This is one rung of a set; all rungs were measured the same way (table below).

Use with mlx-vlm

No fork and no extra code: mlx-vlm 0.7.8 and later include the kolibri1 architecture.

pip install -U "mlx-vlm>=0.7.8"
python -m mlx_vlm generate --model eins78/Kolibri-1-mlx-mixed-4-8bit-vlm --max-tokens 400 \
  --temperature 1.0 --top-p 0.97 --top-k 128 \
  --prompt "Erkläre in zwei Sätzen, warum der Himmel blau ist."

from mlx_vlm import load, generate

model, processor = load("eins78/Kolibri-1-mlx-mixed-4-8bit-vlm")
tokenizer = processor.tokenizer if hasattr(processor, "tokenizer") else processor
messages = [{"role": "user", "content": "Erkläre in zwei Sätzen, warum der Himmel blau ist."}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False, reasoning_effort="low"
)
result = generate(model, processor, prompt, max_tokens=600, temperature=1.0, top_p=0.97, top_k=128)
print(result.text)

Recommended sampling (Aleph Alpha): temperature 1.0, top_p 0.97, top_k 128. The chat template switches reasoning with reasoning_effort ("none", "low", "medium", "high") or enable_thinking=False; with reasoning on the model writes <think>...</think> first. Tool calls come out as Hermes-style <tool_call>{"name": ..., "arguments": ...}</tool_call> (pass tools=[...] to apply_chat_template).

Not for mlx-lm: the weight names follow mlx-vlm's layout (language_model. prefix), so mlx_lm.load does not read these files, even with the open mlx-lm pull request for Kolibri. For mlx-lm, see eins78/Kolibri-1-mlx-mixed-4-8-bit.

The set

Top-1 token agreement / mean KL divergence against the unquantised model in bf16, on two prompts longer than the 513-token window (English 618 tokens, German 612 tokens), scored at every position of the prompt. Higher top-1 and lower KL are better. For scale, bf16 itself against fp32 scores 0.977 / 0.0066 (en_long) and 0.923 / 0.0318 (de_long).

There is no bf16 MLX copy: stock mlx-vlm loads Aleph-Alpha/Kolibri-1-BF16 directly.

Measured quality of this model

Baselines: bf16 is the unquantised BF16 release run in bf16 through mlx-vlm's own layers; fp32 is the same weights run in fp32. The last column shows how far bf16 itself is from fp32.

The short prompts (13 to 24 tokens) are noisy even for bf16: the first positions of a context carry very large activations, and small errors there change the top-1 token more often.

Tested on this machine

Loaded and run with stock mlx-vlm 0.7.8 on an Apple M4 Pro with 64 GB, greedy decoding. Other services were running on the machine, so the speeds are a lower bound.

How this was verified

1. The mlx-vlm model code is exact. mlx-vlm 0.7.8's own kolibri1 layers, run in fp32 one layer at a time from the embedding to the logits on the FP8 checkpoint, reproduce a fp32 PyTorch reference written from Aleph Alpha's vLLM plugin: top-1 1.000 and top-5 1.000 on five German and English prompts, mean KL about 1e-11, identical expert choice at every layer. So every difference measured on this card is precision (bf16 and quantisation), not a wrong implementation. The reference is a torch rebuild of the plugin's math; the plugin itself needs CUDA and was not run.
2. Each rung is measured on its own files against bf16 and fp32 runs of the BF16 weights, through the same mlx-vlm layers.
3. Rungs that fit in 64 GB were also loaded whole with mlx_vlm.load and given the smoke and speed test.

Scripts and method: kolibri-mlx, section 11. Stock mlx-vlm and the MLX release (scripts/vlm_chain.py, scripts/vlm_e2e.py, scripts/vlm_smoke.py, scripts/vlm_convert.py).

AI assistance

The conversion, the verification scripts, the measurements and this model card were produced by an AI coding agent (Claude Code, Claude Opus 5.5) working from a written brief, directed and reviewed by the repository owner (eins78). The numbers on this card were generated from the result files by script, not typed by hand. The upload was done by the agent with the owner's approval.

License

Apache License 2.0, inherited from Aleph Alpha's Kolibri-1-BF16 (the LICENSE file is included). Model weights and tokenizer are by Aleph Alpha; this repository only changes the format and precision. Aleph Alpha publishes no acceptable use policy for Kolibri and asks users to use it lawfully; see the Responsible Use section of the original card.