sarvam-30b — MLX MXFP4

MLX MXFP4 quantization of sarvamai/sarvam-30b, a 32B-parameter Mixture-of-Experts text-generation model from Sarvam AI. Text-only (no vision tower).

sarvam-30b uses a custom sarvam_moe architecture: 19 transformer layers (the first dense, the rest MoE), 128 routed experts + 1 shared expert with top-6 sigmoid-gated routing (DeepSeek-style: expert bias for load balancing, routed_scaling_factor 2.5), grouped-query attention (64 query heads / 4 KV heads) with QK-RMSNorm, and an untied 262K-vocab lm_head (multilingual tokenizer covering Indian languages). Runs on Apple Silicon via mlx-lm.

⚠️ Requires vendoring an unmerged mlx-lm PR. As of this quantization, sarvam_moe is not in any released mlx-lm version — support exists only as an open, unmerged draft, ml-explore/mlx-lm#991. This repo's weights were produced using that PR's model definition, vendored locally. To load this model you must vendor the same file into your own mlx-lm install:

pip install -U mlx-lm
curl -o "$(python -c 'import mlx_lm, os; print(os.path.dirname(mlx_lm.__file__))')/models/sarvam_moe.py" \
  https://raw.githubusercontent.com/ml-explore/mlx-lm/c686bdab1cbf6b5f5364675a8bc858ead97c92eb/mlx_lm/models/sarvam_moe.py

Once PR #991 merges, this step won't be necessary. This will not load in LM Studio (or any tool bundling its own unpatched mlx-lm/mlx-vlm) until then.


Precision	MXFP4 (E2M1 + E8M0 shared scale, group size 32)
Bits per weight	~4.6 bpw (mixed — see below)
On-disk size	17 GB (4 shards)
Quantized	attention projections + all MoE expert weights + `embed_tokens`
Kept full precision (bf16)	`lm_head` (~1.07B params, untied), and each layer's
MoE router weight (`mlp.gate.weight`) — quantizing the router double-damages routing
decisions, so the architecture's own quantization predicate excludes it

Quantizations

Variant	Bits	Size
`sarvam-30b-MXFP4`	~4.6 bpw	17 GB	← this repo
`sarvam-30b-MXFP8`	~8.5 bpw	32 GB	higher fidelity / for 48 GB+

Verification

Smoke-tested on Apple Silicon via mlx-lm with deterministic greedy decoding, inspecting raw token IDs (not just detokenized text) — this matters extra here because embed_tokens (the input embedding table) is quantized, unlike lm_head which the architecture's own quantization predicate keeps full precision.

Tested English, Hindi, and Tamil (this is a multilingual Indian-language model, so an English-only check would miss degradation in non-Latin scripts):

Prompt	Language	Result
"What is the capital of France?"	English	Correct ("Paris"), coherent reasoning
"What is 25 + 17?"	English	Correct step-by-step arithmetic ("42")
"भारत की राजधानी क्या है?" (capital of India)	Hindi	Correct ("New Delhi"), coherent reasoning
"एक पंक्ति में बताइए कि मशीन लर्निंग क्या है।" (explain ML in one line)	Hindi	Coherent, on-topic
"தமிழ்நாட்டின் தலைநகரம் எது?" (capital of Tamil Nadu)	Tamil	Correct ("Chennai"), coherent reasoning

All raw token-ID sequences were clean (no repetition loops or garbage runs).

Usage (mlx-lm)

pip install -U mlx-lm
# then vendor mlx_lm/models/sarvam_moe.py from PR #991 as shown above

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/sarvam-30b-MXFP4")
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))

This is a reasoning model; it emits a <think>...</think> block before its answer — give it enough max_tokens or the visible answer can be truncated while it's still reasoning.

Notes & limitations

Custom architecture, unmerged upstream support. See the vendoring step above — this is a hard requirement, not a suggestion, until PR #991 merges.
Text-only. No vision/image support in the base model.
tie_word_embeddings: false — lm_head is a separate ~1.07B-parameter matrix, kept in bf16 (see table above).
embed_tokens (the input embedding table) is quantized. The MXFP4 smoke test above specifically checked non-English prompts to catch degradation here.
Inherits all capabilities and limitations of the base model. See the original model card.
Quantized by @sahilchachra with MLX. Apache-2.0.