Mistral-Large-4-1T-A52B — EXL3 2.5 bpw (BertholomusAI)
EXL3 quantization of mistralai/Mistral-Large-4-1T-A52B at
revision f12dabdc, made by BertholomusAI: routed experts at 2.5 bits per weight, attention, shared experts and the
dense layer at 5.0, the output head at 6.0. Model and weights by Mistral AI (attribution notice: 2026 - Mistral), under
the Mistral Ecosystem Open-Weight License. This repository holds the quantized weights and the model's own draft head; the original model card, benchmarks and prompt format are on
Mistral AI's model page. The chat template, the tokenizer
files and LICENSE are Mistral AI's, unchanged. This project is not affiliated with or endorsed by Mistral AI.
Made to serve on four NVIDIA DGX Spark (GB10) nodes with the model's full 1,048,576-token context, by
TensorFold (by ashhart) and our mistral_large4 engine family on branch
mistral-large-4-tp4. Recipe, launch steps and
every measurement: bertholomus/mistral-large-4-tensorfold-tp4-4xgb10
(v0.2.0).
License
These weights are a derivative of Mistral Large 4 and stay under the Mistral Ecosystem Open-Weight License, a modified
MIT license, included unchanged as LICENSE. Its notice must go with every copy. Condition 2 matters most: if
you or an affiliate run a Model as a Service business, you must enter into "a separate commercial license with Mistral"
before using the model or its derivative works for any commercial purpose. Internal use is exempt under condition 3,
and the license defines these terms: read it before you use the weights.
Quantization
| Source | mistralai/Mistral-Large-4-1T-A52B, revision f12dabdc0c1e99e7f7924461a6f308859b05af4d |
| Format | EXL3 (exllamav3 1.6.6 by turboderp, mul1 codebook), with our Mistral Large 4 loader |
| Calibration | 128 rows × 2,048 tokens |
| Routed experts (71 layers, 128 each) | 2.5 bits |
| Shared experts, dense MLP (layer 0) | 5.0 bits |
Attention q_proj, o_proj |
5.0 bits |
| Output head | 6.0 bits |
Indexer wq_b |
FP8 with its 128×128 block scales, as in the source |
| Left at source precision (BF16) | the embedding, norms, the router, kv_a_proj_with_mqa, kv_b_proj, the other indexer tensors |
| Draft head | the model's own mtp_dspark.*, 38 BF16 tensors in draft/mtp_dspark.safetensors, the same bytes as the source, not quantized |
| Not included | the vision encoder and its adapters: this pack serves text only |
| Size | 315.2 GiB, and 3.9 GiB for the draft head |
Layout: one EXL3 file and one source-precision file per layer (L<nn>.safetensors, L<nn>-nq.safetensors), then
globals.safetensors, head.safetensors, ml4_pack.json (the pack's manifest), sha256.txt (checksums) and the draft head
in draft/, outside the manifest. It is not a
standard exllamav3 folder: it needs the engine branch above, or a loader that reads this layout.
Quality
What we measured is the serving engine against an independent reference on the same weights: mean next-token loss through our exllamav3 loader and through the engine, on the same 2,047 predicted tokens: 2.5177 against 2.5165 (difference -0.0012 ± 0.0086, two standard errors), with the same top token on 95.1 % of them. Three codes hidden in a prompt were found 3 of 3 at 1,044,249 tokens and 3 of 3 at 120,163 tokens, and our chat and tool-call set passed 6 of 6.
What 2.5 bits costs against the original weights is not measured yet.
Serving (measured, 4× GB10, one build, greedy)
- Decode, one stream, with the draft head: 29.9-30.3 tok/s at 8K, 29.8-30.0 at 32K, 29.2-29.4 at 127K on code (three runs each), 25.3-26.2 on an essay, 37.4-40.8 on filler text, and 21.69 at 1M (one run). Greedy output is the same tokens with the head on or off.
- Prompt: 1,675-1,719 tok/s at 8K, 1,811-1,856 at 32K, 1,751-1,752 at 127K (three runs each). A prompt of 1,044,260 tokens took 803.5 s (13.4 minutes) to its first token.
- Eight streams at once: 64.4 tok/s in all, 8.1 tok/s each (one run).
- A short request beside a long prompt: first token in 1.95-2.73 s beside prompts of 120K and 300K, 2.8-3.0 s about 900,000 tokens into a 1M prompt.
- Two hours of mixed load: 1,379 chats, 13 prompts of 66-124K tokens and 3 of 156-270K, all under load at once: 0 errors, no out-of-memory, no hang.
- Memory: the launch line keeps 5 GiB free on a node and caps the allocator at 103 GiB. Lowest MemAvailable 6.63-6.82 GiB through the rows, 6.80-7.27 GiB through two hours of load.
Not done yet: the conversion code is not published, so this pack can be downloaded but not remade from the original. The recipe has the full list.
Use
hf download bertholomus/Mistral-Large-4-1T-A52B-EXL3-2.5bpw --local-dir Mistral-Large-4-1T-A52B-EXL3-2.5bpw
Each of the four nodes needs the whole pack and the draft head in the same folder, and about 90 GiB more for the cache the first start writes. The recipe has the launch steps.
Credits
Mistral AI made the model. ExLlamaV3 by turboderp gives the EXL3 format and its converter, and TensorFold by ashhart the engine this family plugs into. The draft head is Mistral AI's, under the same license as the model.