mlx-community/flint-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. Try the Lab · All OptiQ quants · Docs

A mixed-precision MLX quant of RecursiveSI/flint, Recursive's 40B-A2.5B mixture-of-experts model for coding agents. 25.6 GB on disk, down from about 80 GB bf16. Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix: sensitive layers get 8 bits, robust ones 4. See the original card for what the model is trained to do and how Recursive evaluates it.

Needs mlx-optiq

Stock mlx-lm has no flint class yet. OptiQ ships an MLX port of the architecture that registers with mlx-lm on import optiq. It was checked against Recursive's SGLang serving kit: learned attention sinks, a 513-token sliding window on 35 of the 49 layers, no positional encoding on the full-attention layers, and sigmoid top-6 routing over 256 experts plus a shared expert.

pip install "mlx-optiq>=0.7.2"

import optiq  # registers the flint arch with mlx-lm
from mlx_lm import load, generate

model, tok = load("mlx-community/flint-OptiQ-4bit")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
    tokenize=False, add_generation_prompt=True,
)
print(generate(model, tok, prompt=prompt, max_tokens=4096))

Flint reasons before it answers, so give it a generous max_tokens. Recursive recommends temperature 1.0, top-p 0.95 and top-k 20. For an OpenAI- and Anthropic-compatible server with tool calls and prompt caching:

optiq serve --model mlx-community/flint-OptiQ-4bit

The weights take 26 GB of unified memory. It was built and scored on a 36 GB Mac, where long prompts leave little headroom; 48 GB or more is comfortable.

Quantization details

Following the naming llama.cpp uses for its mixed quants, the "4bit" label denotes the family, not the weighted average.

Benchmarks

Six-metric Capability Score: the unweighted mean of MMLU, GSM8K, IFEval, BFCL, HumanEval and HashHop. Scored in reasoning mode (generative MMLU and a large thinking budget), the setting that measures an always-on thinking model fairly.

Every metric gets one equal vote. See the eval-framework writeup for the full methodology.

Links

• Project website: mlx-optiq.com
• All OptiQ quants: mlx-optiq.com/models
• PyPI: pypi.org/project/mlx-optiq
• Base model: RecursiveSI/flint