Mistral-Large-4-1T-A52B GGUF

The 1-trillion-parameter Mistral Large 4, in 256 GB or 512 GB of memory.

233 GB and 492 GB builds · lower KL divergence than unsloth's UD files of the same size · llama.cpp

OrcaRouter AI Gateway · X · Discord · GitHub · All Models

---

The full Mistral Large 4, run locally.

These are GGUF builds of Mistral-Large-4-1T-A52B. It is a 1T-parameter MoE with 128 routed experts (top-4) and one shared expert, 52B active per token. It uses MLA attention, DeepSeek-style sparse attention (top-2048) and a 1M-token context.

Routed experts are 98% of the weights, so that is where the bits were spent. Each group of expert layers got its own precision from measured KL divergence, not from a hand-set rule.

At the same file size, both builds sit closer to the original than unsloth's UD-IQ1_S and UD-Q3_K_XL: 20% lower KLD at 233 GB, 13% lower at 492 GB, on held-out text the search never saw.

Vision and speculative decoding included. The vision projector (mmproj) adds image input, and Mistral's DSpark drafter makes generation 1.4–1.5× faster.

---

Pick a build

• Mix-512G fits a 512 GB machine. That covers a 512 GB Mac, a 512 GB server running on CPU, or 4 × H200. It is close to the original: 97% of next tokens match, and perplexity is 0.75% higher.
• Mix-256G fits a 256 GB machine. That covers a 256 GB Mac, a 256 GB PC on CPU, or 2 × H200. It averages under 2 bits per expert weight, which costs quality: 92% of next tokens match, and perplexity is 8.7% higher.

Use Mix-512G if you have the memory. Use Mix-256G to run the full model on half of it.

---

Evaluation

All builds are scored against a Q8_0 GGUF of the official FP8 checkpoint, using llama.cpp's full-vocabulary KLD.

The text is Mistral Large 4's own answers. Prompts come from four sources:

• English chat (UltraChat 200k)
• non-English chat (WildChat)
• code (Magicoder OSS-Instruct)
• math reasoning (OpenR1-Math)

The Q8_0 model answered each one, sampled as the model card recommends (temperature 1.0, top_p 0.95), with reasoning effort drawn per prompt from low / medium / high. Each conversation was rendered through the chat template with its reasoning as a [THINK] block. KLD is counted on the assistant tokens, reasoning and answer, over 64 windows of 2048 tokens. Truncated answers were dropped. The precision search used a separate draw and never saw this text.

Each build is compared with unsloth's dynamic GGUF of the same size, from unsloth/Mistral-Large-4-1T-A52B-GGUF. Their published files were downloaded and scored with the same text, teacher and llama.cpp build.

At the same size, Mix-256G has 20% less KL divergence than UD-IQ1_S, and Mix-512G has 13% less than UD-Q3_K_XL. Both are also a little smaller, and both win in the tail (KLD p99).

Why the model's own text. Mistral Large 4 always reasons before it answers. On other models' answers (no reasoning block, someone else's style) it makes confident misses on ~3% of tokens even at Q8_0, and which of those misses a quant happens to flip dominated the KLD. The same applies to the user turns, which are human-written. So the score covers what the model itself writes.

Method note. These are full-vocabulary KLD and next-token agreement at 2k context, not a downstream benchmark.

---

How the precision was chosen

The budget is set on the bytes that have to fit in memory: 233.0 GB for Mix-256G (the size of UD-IQ1_S) and 492.7 GB for Mix-512G (the size of UD-Q3_K_XL).

1. Measure. For every group of tensors, the search measures the real KLD of moving it one precision level up and one level down. Groups are as follows:

   • routed experts: one projection (gate, up or down) over three layers
   • attention: q, output, v_b and the sparse-attention indexer's q_b, one group each over all layers
   • shared expert: gate, up and down, one group each
   • leading dense FFN: layer 0
   • embedding and output head: one group each

   Each measurement runs llama.cpp's own kernels on 32 (Mix-512G) or 64 (Mix-256G) windows of 2048 tokens of the model's own text.

2. Choose. A knapsack picks the combination of moves that fits the budget exactly.

3. Accept. Only steps whose candidate KLD was actually measured lower are kept.

Mix-256G puts its bits as follows:

Mix-512G puts its bits as follows:

In both builds, the MLA kv_a / k_b projections and the indexer's k projection stay Q8_0, and the router, norms and SSMax parameters stay F32 (0.75 GB in all).

The importance matrix comes from ~740k tokens of the exllamav3 calibration mix:

• web text (C4)
• code
• multilingual text
• technical text
• Wikipedia
• tiny stories
• random token rows

None of it overlaps the evaluation text.

---

What's in this repo

Mix-256G is split into 6 parts and Mix-512G into 12, each under 45 GB. The chat template is embedded in each build.

The vision projector and the DSpark drafter both come from the official checkpoint. The projector is kept at BF16. The drafter is Mistral's 3-layer DSpark head, which proposes 3 tokens at a time and is stored at Q8_0. Both work with either build.

---

Serving (llama.cpp)

Mistral Large 4 support landed in llama.cpp master with ggml-org/llama.cpp#31376 (commit 78feec0). These files were made and tested at master commit e0f142d:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout e0f142d   # or any later master
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-cli
# Mac: cmake -B build (Metal is on by default) · CPU only: cmake -B build

hf download orcarouter/Mistral-Large-4-1T-A52B-GGUF --include "Mix-256G/*" "mmproj-*" "*DSpark*" \
  --local-dir ./ml4-gguf

# 2 x H200 (Mix-256G) -- llama.cpp spreads the layers across the GPUs by free memory
llama-server -m ml4-gguf/Mix-256G/Mistral-Large-4-1T-A52B-Mix-256G-00001-of-00006.gguf \
  -c 32768 --jinja -fitt 4096

# 4 x H200 (Mix-512G)
llama-server -m ml4-gguf/Mix-512G/Mistral-Large-4-1T-A52B-Mix-512G-00001-of-00012.gguf \
  -c 32768 --jinja

# 2 x H200 (Mix-256G) with image input and DSpark speculative decoding; the same two flags work with Mix-512G
llama-server -m ml4-gguf/Mix-256G/Mistral-Large-4-1T-A52B-Mix-256G-00001-of-00006.gguf \
  -c 32768 --jinja -fitt 4096 \
  --mmproj ml4-gguf/mmproj-Mistral-Large-4-1T-A52B-BF16.gguf \
  -md ml4-gguf/Mistral-Large-4-1T-A52B-DSpark-Q8_0.gguf --spec-type draft-dspark --spec-draft-n-max 3

# CPU only, 256 GB RAM (not tested)
llama-server -m ml4-gguf/Mix-256G/Mistral-Large-4-1T-A52B-Mix-256G-00001-of-00006.gguf \
  -ngl 0 -c 16384 --jinja

• Reasoning is always on. Pick the effort per request with "chat_template_kwargs": {"reasoning_effort": "low" | "medium" | "high"} (default high). The reasoning comes back in message.reasoning_content, and the answer in message.content.
• Sampling. Mistral recommends temperature=1.0, top_p=0.95 for every reasoning effort. The files carry these as their default sampling metadata.
• Point llama.cpp at the 00001 part. It finds the other parts.
• Mix-256G on exactly 2 x H200. Keep -fitt 4096. With the default 1 GB margin per GPU, the first long prompt can run out of GPU memory.
• Images. With --mmproj, send OpenAI-style image_url content parts (base64 data URLs work). On both builds, the model read a newspaper front page (publication, date and headline) and counted and named the shapes and text in a synthetic image correctly, with and without DSpark.
• DSpark. The draft proposes 3 tokens per step and the main model checks them, so every token is still the main model's choice. At temperature 0, outputs are not bit-identical to plain decoding: checking a block in one batch changes the arithmetic slightly, so near-ties can break the other way.
• Mac. macOS lets the GPU wire only about 75% of RAM by default. Raise the limit before loading on Metal, for example sudo sysctl iogpu.wired_limit_mb=500000 for Mix-512G on a 512 GB Mac, and close other apps.

Measured with llama-bench (512-token prompt, 128 generated tokens, one request at a time):

With the DSpark drafter, measured through llama-server on six prompts (code, explanation, math, story, translation, SQL), up to 1024 generated tokens each, low reasoning effort, one request at a time:

The gain depends on the text: 1.2–1.4× on long prose (the story and the explanation), 1.6–1.7× on code and math. CPU and Apple Metal speeds have not been measured yet.

---

Model details

---

Bias, risks, and limitations

• Mix-256G is noticeably degraded. Under 2 bits per expert weight is a large cut for any model. It is the best we found at this size, not a match for the original.
• Tested on CUDA only. CPU and Apple Metal have not been run with these files yet.
• Vision was checked on a few images, not benchmarked. Long-context behaviour was not tested: all measurements are at 2k context or on short chats.
• Inherits the biases and limitations of Mistral-Large-4-1T-A52B.
• Requires llama.cpp with PR #31376 (master 78feec0 or later). Tools built on older llama.cpp, such as older Ollama and LM Studio releases, will not load these files.

License

The Mistral Ecosystem Open-Weight License, inherited from the base model mistralai/Mistral-Large-4-1T-A52B. It is a modified MIT license that allows commercial and non-commercial use. Operating a Model-as-a-Service business with the model requires a separate commercial license from Mistral. See the license for the terms.

---

One Gateway. Every Model.

Route Smarter · Ship Safer · Spend Less

Website · Models · GitHub · Discord · X