MLX MXFP8 quantization of sarvamai/sarvam-30b, a 32B-parameter
Mixture-of-Experts text-generation model from Sarvam AI. Text-only (no vision tower).
sarvam-30b uses a custom sarvam_moe architecture: 19 transformer layers (the first
dense, the rest MoE), 128 routed experts + 1 shared expert with top-6 sigmoid-gated routing
(DeepSeek-style: expert bias for load balancing, routed_scaling_factor 2.5), grouped-query
attention (64 query heads / 4 KV heads) with QK-RMSNorm, and an untied 262K-vocab
lm_head (multilingual tokenizer covering Indian languages). Runs on Apple Silicon via
mlx-lm.
⚠️ Requires vendoring an unmerged mlx-lm PR. As of this quantization, sarvam_moe is
not in any released mlx-lm version — support exists only as an open, unmerged draft,
ml-explore/mlx-lm#991. This repo's weights
were produced using that PR's model definition, vendored locally. To load this model you
must vendor the same file into your own mlx-lm install:
pip install -U mlx-lm
curl -o "$(python -c 'import mlx_lm, os; print(os.path.dirname(mlx_lm.__file__))')/models/sarvam_moe.py" \
https://raw.githubusercontent.com/ml-explore/mlx-lm/c686bdab1cbf6b5f5364675a8bc858ead97c92eb/mlx_lm/models/sarvam_moe.py
Once PR #991 merges, this step won't be necessary. This will not load in LM Studio (or any tool bundling its own unpatched mlx-lm/mlx-vlm) until then.
| Precision | MXFP8 (E4M3 + E8M0 shared scale, group size 32) |
| Bits per weight | ~8.5 bpw (mixed — see below) |
| On-disk size | 32 GB (30 shards) |
| Quantized | attention projections + all MoE expert weights + embed_tokens |
| Kept full precision (bf16) | lm_head (~1.07B params, untied), and each layer's |
MoE router weight (mlp.gate.weight) — quantizing the router double-damages routing |
|
| decisions, so the architecture's own quantization predicate excludes it |
| Variant | Bits | Size | |
|---|---|---|---|
sarvam-30b-MXFP4 |
~4.6 bpw | 17 GB | smaller / for 32 GB+ |
sarvam-30b-MXFP8 |
~8.5 bpw | 32 GB | ← this repo |
This higher-fidelity build was verified structurally: correct tensor shapes,
format: mlx metadata, consistent shard index, lm_head and each layer's MoE router
(mlp.gate.weight) correctly left unquantized (same predicate as the MXFP4 build).
Full token-by-token generation was not benchmarked on the 26 GB test machine because this
build's 32 GB exceeds available RAM; on a 48 GB+ Mac it runs at normal speed. Since MXFP8
uses more bits than the MXFP4 build — which passed the full English/Hindi/Tamil
generation smoke test above — it is at least as faithful to the base model.
See the MXFP4 build for the full generation smoke-test results.
pip install -U mlx-lm
# then vendor mlx_lm/models/sarvam_moe.py from PR #991 as shown above
from mlx_lm import load, generate
model, tokenizer = load("sahilchachra/sarvam-30b-MXFP8")
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))
This is a reasoning model; it emits a <think>...</think> block before its answer — give it
enough max_tokens or the visible answer can be truncated while it's still reasoning.
tie_word_embeddings: false — lm_head is a separate ~1.07B-parameter matrix, kept in
bf16 (see table above).embed_tokens (the input embedding table) is quantized. The MXFP4 smoke test above
specifically checked non-English prompts to catch degradation here.model-00029-of-00030.safetensors 2.1 GBmodel-00014-of-00030.safetensors 1.2 GBmodel-00001-of-00030.safetensors 1.2 GBmodel-00023-of-00030.safetensors 1.2 GBmodel-00005-of-00030.safetensors 1.2 GBmodel-00026-of-00030.safetensors 1.2 GBmodel-00016-of-00030.safetensors 1.2 GBmodel-00008-of-00030.safetensors 1.2 GBmodel-00021-of-00030.safetensors 1.2 GBmodel-00012-of-00030.safetensors 1.2 GBmodel-00003-of-00030.safetensors 1.2 GBmodel-00011-of-00030.safetensors 1.2 GBmodel-00020-of-00030.safetensors 1.2 GBmodel-00024-of-00030.safetensors 1.2 GBmodel-00006-of-00030.safetensors 1.2 GBmodel-00027-of-00030.safetensors 1.1 GBmodel-00017-of-00030.safetensors 1.1 GBmodel-00009-of-00030.safetensors 1.1 GBmodel-00018-of-00030.safetensors 1.1 GBmodel-00010-of-00030.safetensors 1.1 GBmodel-00030-of-00030.safetensors 1.1 GBmodel-00019-of-00030.safetensors 1.1 GBmodel-00015-of-00030.safetensors 1.1 GBmodel-00022-of-00030.safetensors 1.1 GBmodel-00025-of-00030.safetensors 1.1 GBmodel-00004-of-00030.safetensors 1.1 GBmodel-00007-of-00030.safetensors 1.1 GBmodel-00013-of-00030.safetensors 1.1 GBmodel-00002-of-00030.safetensors 689 MBmodel-00028-of-00030.safetensors 537 MB有文件？提交种子。
提交种子sarvamai/sarvam-30b