MiniCPM-V-4.7-35B-A3B for mlx-serve (mixed 4/8-bit)
An MLX affine mixed-precision quantization of openbmb/MiniCPM-V-4.7-35B-A3B, distributed under the base model's Apache-2.0 license.
Text-only pack
This pack contains language model weights only. It does not support image or video inputs and does not include the MiniCPM vision tower or merger. Its native root model type is minicpmv4_7, and it requires mlx-serve v26.10.2 or later.
Quantization
Affine quantization with group size 64: routed MoE experts are 4-bit; attention, router gates, shared-expert gates, shared experts, token embeddings, and LM head are 8-bit. Norms, scales, and biases remain BF16.
Setup
mlx-serve serve --model pgmharikrishnan/MiniCPM-V-4.7-35B-A3B-MLX-Serve-mixed-4-8bit
Benchmark
Standard suite on Apple M4 Max with mlx-serve 26.10.2:
| Benchmark | Result |
|---|---|
| MMLU-Pro (400 questions) | 90.00% (360/400) |
| GSM8K (50 questions) | 100% |
| Coding (6 tasks) | 6/6 |
| Decode | 47.8 tok/s |
| Prefill | 59.6 tok/s |
| Peak memory | 27.7 GB |
MMLU-Pro is public, so training contamination cannot be ruled out.
Compared with the 4-bit pack
On this run, the 4-bit pack decoded and prefilled faster (60.6 vs 47.8 tok/s; 343 vs 59.6 tok/s) and used less peak memory (24.7 vs 27.7 GB). MMLU-Pro was 89.75% for 4-bit and 90.00% for mixed; both scored 100% on GSM8K and 6/6 on coding.
Attribution
MiniCPM-V 4.7 was created by OpenBMB. This is an independent MLX conversion for mlx-serve.
Disclaimer
This is an unofficial, text-only conversion. Validate it for your workload before production use.