Ministral-3-3B-Instruct-2512 · GGUF

Native image input, multilingual chat and tool calling in a 3B-class model.

GGUF builds of Ministral-3-3B-Instruct-2512-BF16 by Mistral AI for llama.cpp and SoAI, calibrated on real chat, code and tool calls.

---

At a glance

Choosing a file

[!TIP] Q4_K_M suits most machines. It is about a third of the size of 16-bit weights and stays close to them in quality. If Q8_0 fits in your memory, its output stays closest to the original. SoAI does not publish quants below 4 bits, because what they save in space rarely makes up for their quality loss.

Running it takes memory for the file and for its context, which grows with the context length you set. llama.cpp can run this model on GPU, on CPU, or split its layers between both.

Images need the mmproj-Ministral-3-3B-Instruct-2512-f16.gguf add-on as well, loaded with --mmproj. llama-server -hf downloads it automatically.

With one 32K context and an FP16 KV cache on an NVIDIA RTX A4000, Q4_K_M used 6,660 MiB (7.0 GB) of GPU memory after text and image requests, including its image encoder. The launch examples allocate 32K; increase -c for longer conversations.

Quantization comparison against BF16 on held-out text, using 100 chunks of 512 tokens:

Quick start

With llama.cpp, this command starts a local server with a built-in chat UI and an OpenAI-compatible API:

llama-server -hf SoAIHQ/Ministral-3-3B-Instruct-2512-GGUF:Q4_K_M -c 32768 --temp 0.05

Open http://localhost:8080 to chat, or send requests to the API:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Explain GGUF in one sentence."}]
}'

Download the files to a folder first

hf download SoAIHQ/Ministral-3-3B-Instruct-2512-GGUF --include "Ministral-3-3B-Instruct-2512-Q4_K_M.gguf" "mmproj-*" --local-dir Ministral-3-3B-Instruct-2512-GGUF
llama-server -m Ministral-3-3B-Instruct-2512-GGUF/Ministral-3-3B-Instruct-2512-Q4_K_M.gguf -c 32768 --temp 0.05 \
  --mmproj Ministral-3-3B-Instruct-2512-GGUF/mmproj-Ministral-3-3B-Instruct-2512-f16.gguf

Recommended settings

Mistral AI recommends these values in the original model card. The launch examples use 0.05. Set temperature per request when a task calls for more varied answers.

Agents and tool calling

Send each assistant tool call back with its original id, type and function, then append a tool message with a matching tool_call_id. Keep earlier calls and results in message history so the model can continue from search to reading a document and answering.

How these files are made

Q4_K_M is quantized with an importance matrix (imatrix), which tells llama.cpp which weights matter most for output so it can store them with more precision. SoAI computes this imatrix from its own calibration corpus instead of generic web text, and formats every conversation in it with this model's chat template, including its native tool-call syntax. Calibration therefore measures token patterns this model sees in use: multi-turn chat in 21 languages, code edits in 22 programming languages, tool calls and their results, step-by-step math, and web prose.

Q8_0 is quantized without the imatrix, because llama.cpp's Q8_0 format does not use one.

Before any quantization, the build checks that chat, thinking and tool-call markers are stored as special tokens and that its embedded chat template matches Mistral AI's. A conversion that imports those markers as plain text silently breaks turns, thinking and tool calls, so in that case the build stops instead of publishing anything.

This repository also publishes its imatrix file, and the provenance table below pins the original revision.

Calibration corpus sources All files and SHA-256 checksums

The full checksums are in SHA256SUMS.

Provenance

License and credits

These files are distributed under the original model's license (apache-2.0), and all of its terms continue to apply. The license and original model card also cover this model's capabilities, training data, benchmarks and limitations. A copy of the license text is included in this repository as LICENSE.

Mistral AI designed and trained this model, and SoAI only repackages its published weights. Conversion uses llama.cpp by Georgi Gerganov and ggml contributors.

Run it with SoAI

Download SoAI for Windows, macOS, Linux or Android to manage models and use them through chat or an API.

_{Packaged by SoAI.}