Mistral Large 4 — EXL3 3.0 bpw

A community EXL3 conversion of Mistral Large 4 — Le Chonk, prepared by Piotr Kosecki for local inference on four NVIDIA GB10 / DGX Spark class machines.

The weights are complete: 76 Safetensors shards, 403.28 GB (decimal), with configuration and tokenizer files. The included experimental TensorFold recipe runs text generation with 8192 tokens of context, one active request, and no speculative decoding.

→ Four-Spark setup and launch recipe — includes the required custom runtime, checksums, configuration generator and source patch.

→ Original Mistral model card — copied unchanged from the source revision, with Mistral's model description, upstream benchmarks and usage guidance. Its FP8/NVFP4 examples and advertised context capacity describe the original model; use the EXL3-specific recipe above for these files.

If this conversion and the Spark work are useful to you, you can buy me a coffee. Thank you for supporting the time, hardware and experiments behind this release.

Quantization and provenance

“3.0 bpw” is the quantization target for the main matrices, not a claim that every tensor is stored at three bits. Protected tensors, the head, auxiliaries and format overhead contribute to the actual size.

The module-based shard layout avoids another upload of the same weight data just to change shard boundaries. Finalization compared every packaged tensor's dtype, shape and SHA-256 with the compiled conversion, and retained the remaining auxiliary tensors verbatim. The release check matched all 76 remote weight-file hashes and sizes to the frozen package. See validation-summary.json.

What has been checked

• A matched 8188-token internal perplexity check: source FP8 7.9321, EXL3 candidate 8.0660, a 1.69% relative increase. This was measured with the custom ExLlamaV3 implementation and Q8 latent cache; it is a small holdout, not a broad benchmark.
• Native TensorFold text generation and coding checks across four GB10 machines. The frozen plain-decoding comparison measured approximately 11.87 tok/s for code generation and 10.14 tok/s for code editing, at a 2 GHz GPU clock cap, context 8192 and concurrency 1. These are two internal workloads, not a throughput guarantee.
• Configuration identity, complete tensor indexing and equivalence between the published module shards and the compiled layout used in those GPU checks. The public download layout has not been separately rerun through a full GPU benchmark.

The supplied runtime is a community TensorFold fork, not an upstream compatibility claim. Official TensorFold, vLLM, SGLang and Transformers are not drop-in loaders for this particular EXL3 package.

The included recipe is text-only. Preservation of vision weights does not establish working image inference. 262k/1M context, a 524k shared cache, four simultaneous users, and 20 tok/s have not been qualified with this recipe. Its current native cache layout does not fit the previously planned 262k context on four GB10 hosts. No DFlash, DSpark, MTP, draft-model download or original FP8 checkpoint is required by the recipe.

Download

Install the Hugging Face CLI, then download to a filesystem with at least 410 GB free for the model; allow additional room for runtime files, the container image and normal operating margin:

hf download piotrkosecki/Mistral-Large-4-1T-A52B-EXL3-3.0bpw \
  --local-dir /mnt/models/Mistral-Large-4-1T-A52B-EXL3-3.0bpw

Use a shared NAS mounted at the configured path on all four hosts, or identical local copies. Follow the recipe to configure the fabric, copy the small runtime bundle and start the local OpenAI-compatible endpoint.

Acknowledgements

Thank you to Mistral AI and the Mistral Large 4 team for creating and releasing the original model and its documentation; turboderp and the ExLlamaV3 contributors for EXL3 and the quantization foundations; ashhart and the TensorFold contributors for the inference engine, native CUDA kernels and multi-host foundations; and the xgrammar, Hugging Face and NVIDIA communities for the tooling used in this work.

Conversion, local Large4 integration, packaging and four-Spark experiments: Piotr Kosecki, with AI-assisted development and testing. This is an independent community release; the upstream authors have not endorsed its changes or results.

License

The model and original model documentation retain Mistral's Mistral Ecosystem Open-Weight License, including its original attribution and conditions. Read the license for the terms that apply to your use. The custom TensorFold runtime and source modifications are distributed under Apache-2.0 with the upstream notices and third-party licenses in recipes/4x-spark/licenses. The runtime license does not replace the model license.