sarvam-30b — MLX MXFP8

MLX MXFP8 quantization of sarvamai/sarvam-30b, a 32B-parameter Mixture-of-Experts text-generation model from Sarvam AI. Text-only (no vision tower).

sarvam-30b uses a custom sarvam_moe architecture: 19 transformer layers (the first dense, the rest MoE), 128 routed experts + 1 shared expert with top-6 sigmoid-gated routing (DeepSeek-style: expert bias for load balancing, routed_scaling_factor 2.5), grouped-query attention (64 query heads / 4 KV heads) with QK-RMSNorm, and an untied 262K-vocab lm_head (multilingual tokenizer covering Indian languages). Runs on Apple Silicon via mlx-lm.

⚠️ Requires vendoring an unmerged mlx-lm PR. As of this quantization, sarvam_moe is not in any released mlx-lm version — support exists only as an open, unmerged draft, ml-explore/mlx-lm#991. This repo's weights were produced using that PR's model definition, vendored locally. To load this model you must vendor the same file into your own mlx-lm install:

pip install -U mlx-lm
curl -o "$(python -c 'import mlx_lm, os; print(os.path.dirname(mlx_lm.__file__))')/models/sarvam_moe.py" \
  https://raw.githubusercontent.com/ml-explore/mlx-lm/c686bdab1cbf6b5f5364675a8bc858ead97c92eb/mlx_lm/models/sarvam_moe.py

Once PR #991 merges, this step won't be necessary. This will not load in LM Studio (or any tool bundling its own unpatched mlx-lm/mlx-vlm) until then.


Precision	MXFP8 (E4M3 + E8M0 shared scale, group size 32)
Bits per weight	~8.5 bpw (mixed — see below)
On-disk size	32 GB (30 shards)
Quantized	attention projections + all MoE expert weights + `embed_tokens`
Kept full precision (bf16)	`lm_head` (~1.07B params, untied), and each layer's
MoE router weight (`mlp.gate.weight`) — quantizing the router double-damages routing
decisions, so the architecture's own quantization predicate excludes it

Quantizations

Variant	Bits	Size
`sarvam-30b-MXFP4`	~4.6 bpw	17 GB	smaller / for 32 GB+
`sarvam-30b-MXFP8`	~8.5 bpw	32 GB	← this repo

Verification

This higher-fidelity build was verified structurally: correct tensor shapes, format: mlx metadata, consistent shard index, lm_head and each layer's MoE router (mlp.gate.weight) correctly left unquantized (same predicate as the MXFP4 build). Full token-by-token generation was not benchmarked on the 26 GB test machine because this build's 32 GB exceeds available RAM; on a 48 GB+ Mac it runs at normal speed. Since MXFP8 uses more bits than the MXFP4 build — which passed the full English/Hindi/Tamil generation smoke test above — it is at least as faithful to the base model.

See the MXFP4 build for the full generation smoke-test results.

Usage (mlx-lm)

pip install -U mlx-lm
# then vendor mlx_lm/models/sarvam_moe.py from PR #991 as shown above

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/sarvam-30b-MXFP8")
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))

This is a reasoning model; it emits a <think>...</think> block before its answer — give it enough max_tokens or the visible answer can be truncated while it's still reasoning.

Notes & limitations

Custom architecture, unmerged upstream support. See the vendoring step above — this is a hard requirement, not a suggestion, until PR #991 merges.
Text-only. No vision/image support in the base model.
tie_word_embeddings: false — lm_head is a separate ~1.07B-parameter matrix, kept in bf16 (see table above).
embed_tokens (the input embedding table) is quantized. The MXFP4 smoke test above specifically checked non-English prompts to catch degradation here.
Inherits all capabilities and limitations of the base model. See the original model card.
Quantized by @sahilchachra with MLX. Apache-2.0.