Mistral-Large-4-1T-A52B — EXL3 2.5 bpw (BertholomusAI)

EXL3 quantization of mistralai/Mistral-Large-4-1T-A52B at revision f12dabdc, made by BertholomusAI: routed experts at 2.5 bits per weight, attention, shared experts and the dense layer at 5.0, the output head at 6.0. Model and weights by Mistral AI (attribution notice: 2026 - Mistral), under the Mistral Ecosystem Open-Weight License. This repository holds the quantized weights and the model's own draft head; the original model card, benchmarks and prompt format are on Mistral AI's model page. The chat template, the tokenizer files and LICENSE are Mistral AI's, unchanged. This project is not affiliated with or endorsed by Mistral AI.

Made to serve on four NVIDIA DGX Spark (GB10) nodes with the model's full 1,048,576-token context, by TensorFold (by ashhart) and our mistral_large4 engine family on branch mistral-large-4-tp4. Recipe, launch steps and every measurement: bertholomus/mistral-large-4-tensorfold-tp4-4xgb10 (v0.2.0).

License

These weights are a derivative of Mistral Large 4 and stay under the Mistral Ecosystem Open-Weight License, a modified MIT license, included unchanged as LICENSE. Its notice must go with every copy. Condition 2 matters most: if you or an affiliate run a Model as a Service business, you must enter into "a separate commercial license with Mistral" before using the model or its derivative works for any commercial purpose. Internal use is exempt under condition 3, and the license defines these terms: read it before you use the weights.

Quantization


Source	`mistralai/Mistral-Large-4-1T-A52B`, revision `f12dabdc0c1e99e7f7924461a6f308859b05af4d`
Format	EXL3 (exllamav3 1.6.6 by turboderp, mul1 codebook), with our Mistral Large 4 loader
Calibration	128 rows × 2,048 tokens
Routed experts (71 layers, 128 each)	2.5 bits
Shared experts, dense MLP (layer 0)	5.0 bits
Attention `q_proj`, `o_proj`	5.0 bits
Output head	6.0 bits
Indexer `wq_b`	FP8 with its 128×128 block scales, as in the source
Left at source precision (BF16)	the embedding, norms, the router, `kv_a_proj_with_mqa`, `kv_b_proj`, the other indexer tensors
Draft head	the model's own `mtp_dspark.*`, 38 BF16 tensors in `draft/mtp_dspark.safetensors`, the same bytes as the source, not quantized
Not included	the vision encoder and its adapters: this pack serves text only
Size	315.2 GiB, and 3.9 GiB for the draft head

Layout: one EXL3 file and one source-precision file per layer (L<nn>.safetensors, L<nn>-nq.safetensors), then globals.safetensors, head.safetensors, ml4_pack.json (the pack's manifest), sha256.txt (checksums) and the draft head in draft/, outside the manifest. It is not a standard exllamav3 folder: it needs the engine branch above, or a loader that reads this layout.

Quality

What we measured is the serving engine against an independent reference on the same weights: mean next-token loss through our exllamav3 loader and through the engine, on the same 2,047 predicted tokens: 2.5177 against 2.5165 (difference -0.0012 ± 0.0086, two standard errors), with the same top token on 95.1 % of them. Three codes hidden in a prompt were found 3 of 3 at 1,044,249 tokens and 3 of 3 at 120,163 tokens, and our chat and tool-call set passed 6 of 6.

What 2.5 bits costs against the original weights is not measured yet.

Serving (measured, 4× GB10, one build, greedy)

Decode, one stream, with the draft head: 29.9-30.3 tok/s at 8K, 29.8-30.0 at 32K, 29.2-29.4 at 127K on code (three runs each), 25.3-26.2 on an essay, 37.4-40.8 on filler text, and 21.69 at 1M (one run). Greedy output is the same tokens with the head on or off.
Prompt: 1,675-1,719 tok/s at 8K, 1,811-1,856 at 32K, 1,751-1,752 at 127K (three runs each). A prompt of 1,044,260 tokens took 803.5 s (13.4 minutes) to its first token.
Eight streams at once: 64.4 tok/s in all, 8.1 tok/s each (one run).
A short request beside a long prompt: first token in 1.95-2.73 s beside prompts of 120K and 300K, 2.8-3.0 s about 900,000 tokens into a 1M prompt.
Two hours of mixed load: 1,379 chats, 13 prompts of 66-124K tokens and 3 of 156-270K, all under load at once: 0 errors, no out-of-memory, no hang.
Memory: the launch line keeps 5 GiB free on a node and caps the allocator at 103 GiB. Lowest MemAvailable 6.63-6.82 GiB through the rows, 6.80-7.27 GiB through two hours of load.

Not done yet: the conversion code is not published, so this pack can be downloaded but not remade from the original. The recipe has the full list.

Use

hf download bertholomus/Mistral-Large-4-1T-A52B-EXL3-2.5bpw --local-dir Mistral-Large-4-1T-A52B-EXL3-2.5bpw

Each of the four nodes needs the whole pack and the draft head in the same folder, and about 90 GiB more for the cache the first start writes. The recipe has the launch steps.

Credits

Mistral AI made the model. ExLlamaV3 by turboderp gives the EXL3 format and its converter, and TensorFold by ashhart the engine this family plugs into. The draft head is Mistral AI's, under the same license as the model.