Kolibri-1 MLX 6-bit

A 6-bit MLX conversion of Aleph-Alpha/Kolibri-1 for Apple Silicon, published by audreyt. Aleph Alpha trained the original model.

What changed

• Routed experts, the shared expert, attention projections, the embedding, and the output head use MLX affine 6-bit quantization with group size 64. The overall rate is 6.506 bits per weight.
• The 50 MoE routers (mlp.gate) keep their original BF16 weights. The routing biases (expert_bias) and all norms also stay BF16, as in the release.
• The source is the official block-FP8 release. Each FP8 block was dequantized to BF16 and quantized once to 6-bit, with no intermediate format.

This is a conversion of already-quantized FP8 weights, not a quantization directly from the BF16 training checkpoint.

How it differs from other Kolibri-1 conversions

Conversion measurements

Conversion ran on an Apple M5 Max with 128 GB of unified memory.

Inference smoke test

Same Mac, the mlx-lm build above, and the sampling settings recommended by Aleph Alpha (temperature=1.0, top_p=0.97, top_k=128):

Peak memory during generation was 63.6 GB.

These are smoke tests, not benchmark scores. Long-context behavior, tool calling, and quality relative to the original model have not been measured.

Requirements

The kolibri1 architecture is not yet part of a released mlx-lm. Install mlx-lm from the here-be-dragons fork, pinned to the build used for this conversion:

pip install mlx "git+https://github.com/here-be-dragons-ai/mlx-lm@5bc45adb8733f9af29c1c03e9ea947d155328b75"

• About 64 GB of free unified memory. A 96 GB or 128 GB Mac is the practical minimum.
• The weights use mlx-lm key names (no language_model. prefix), so mlx-vlm and its server do not load this repository.

Usage

from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("audreyt/Kolibri-1-MLX-6bit")
messages = [{"role": "user", "content": "Erkläre kurz, was ein Mixture-of-Experts-Modell ist."}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, reasoning_effort="low"
)
sampler = make_sampler(temp=1.0, top_p=0.97, top_k=128)
for r in stream_generate(model, tokenizer, prompt, max_tokens=2048, sampler=sampler):
    print(r.text, end="", flush=True)

reasoning_effort accepts none, low, medium, or high. The original release says the model does not reason unless it is set. A default mlx_lm.generate run on this build, which did not pass reasoning_effort, still opened with a <think> block (873 tokens at 88 tok/s).

Recent transformers releases warn that this tokenizer has an incorrect regex pattern and suggest fix_mistral_regex=True. On German, English, code, and CJK samples, both settings produced identical token IDs.

Reproduce the conversion

hf download Aleph-Alpha/Kolibri-1 \
  --revision e52eb4627d11516b0c01de49210ab5a4e4061444 \
  --local-dir kolibri-fp8-source

pip install mlx "git+https://github.com/here-be-dragons-ai/mlx-lm@5bc45adb8733f9af29c1c03e9ea947d155328b75"

mlx_lm.convert --hf-path kolibri-fp8-source --mlx-path Kolibri-1-MLX-6bit \
  -q --q-bits 6 --q-group-size 64

Use plain -q. The model's own quantization rule is what keeps the routers in BF16. The --quant-predicate mixed_* recipes replace that rule, so they would quantize the routers too.

Dependencies and provenance

• Original model
• Official vLLM plugin, the reference implementation the MLX port follows
• MLX and the here-be-dragons mlx-lm port

The original model's Apache-2.0 license is preserved in LICENSE.