Flint-MLX-Serve-4bit

⚠️ Not recommended: this uniform 4-bit pack fails our fidelity check (61.3% top-1 agreement vs BF16; threshold 85%). Use pgmharikrishnan/Flint-MLX-Serve-mixed-4-8bit instead (87.1%).

An Apple-Silicon MLX pack of Recursive Flint, a 39.8B-total / 2.5B-active coding-agent MoE.

Pack size: ~24 GB. Recommended RAM: 32 GB+ unified memory. Requires mlx-serve with Flint support (in progress).

Recipe

Uniform affine 4-bit quantization, group size 64; norms, vectors, and matrices with fewer than 32 rows remain BF16.

The checkpoint was converted shard-by-shard on CPU from RecursiveSI/flint, retaining the upstream Apache-2.0 license.

mlx-serve quickstart

mlx-serve --model pgmharikrishnan/Flint-MLX-Serve-4bit

OpenAI-compatible request:

curl http://127.0.0.1:11234/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"pgmharikrishnan/Flint-MLX-Serve-4bit","messages":[{"role":"user","content":"Write a Python hello world."}]}'

Results

Teacher-forced fidelity vs BF16 (256 tokens of prose): top-1 agreement 61.3% (pass threshold ≥85%), mean full-vocabulary KL 0.73 nats. ❌ Fails. The mixed 4-8bit pack keeps those at 8-bit and passes.

mlx-serve standard benchmark: pending GPU-locked run.