LFM2.5-24B-A2B for mlx-serve (mixed 4-bit / 8-bit)
An affine MLX quantization of LiquidAI/LFM2.5-24B-A2B. It carries the source repository's LFM 1.0 license unchanged.
This is a text-only model. Its native architecture is lfm2_moe (Lfm2MoeForCausalLM), which is supported natively by mlx-serve v26.10.2 and later; it is not a Qwen-engine conversion.
Benchmark results
Recorded comparison with LiquidAI/LFM2.5-24B-A2B-MLX-4bit:
| Metric | Stock 4-bit | Mixed 4/8-bit | Delta |
|---|---|---|---|
| Plain decode | 85.68 tok/s | 189.75 tok/s | +121.5% (2.21×) |
| Plain prefill | 409.71 tok/s | 733.09 tok/s | +78.9% |
| Best decode | 180.45 tok/s | 184.95 tok/s | +2.5% |
| Best prefill | 482.32 tok/s | 460.79 tok/s | -4.5% |
Long-context ladder
| Rung | Stock prefill / decode | Mixed prefill / decode | Needle |
|---|---|---|---|
| 4K | 1222 tok/s / 128 tok/s | 2644 tok/s / 143 tok/s | Passed / Passed |
| 16K | 1263 tok/s / 84 tok/s | 2441 tok/s / 104 tok/s | Passed / Passed |
| 32K | 1082 tok/s / 67 tok/s | 2156 tok/s / 79 tok/s | Passed / Passed |
| 64K | 866 tok/s / 31 tok/s | 1674 tok/s / 52 tok/s | Passed / Passed |
| 128K | — | — | Both packs failed at the 131,072-token ceiling. |
Quantization
- Routed MoE experts: affine 4-bit, group size 64.
- Trunk projections (attention, convolution, and dense MLP layers): affine 8-bit, group size 64.
- Token embeddings: affine 6-bit, group size 64.
- Router gates and norms: BF16; expert-selection bias: float32.
Usage
Requires mlx-serve v26.10.2 or later.
mlx-serve serve --model pgmharikrishnan/LFM2.5-24B-A2B-MLX-Serve-mixed-4-8bit