MiniCPM5-2B GGUF

Community GGUF quantizations of openbmb/MiniCPM5-2B.

☕ If this GGUF made your day easier, a coffee would make mine.
Send a coffee ☕
I build and test these releases myself. Your coffee helps keep me going.
Thank you for supporting this work.

About MiniCPM5-2B

MiniCPM5-2B is OpenBMB's dense 2B-class model for local assistants, coding agents, tool-use workflows, reasoning, and resource-constrained deployment. It uses the standard LlamaForCausalLM architecture, has 2,516,756,480 parameters, 42 layers, GQA with 16 query / 2 KV heads, and a native 131,072-token context length. See the official model card for the upstream capabilities, evaluation results, datasets, and supported ecosystem.

This is a quantization-only release. No training, fine-tuning, merging, or weight modification other than BF16-to-GGUF conversion and quantization was performed. The source BF16 checkpoint was converted directly from the locked upstream revision; no quantized file was used as input to another quantization rung.

The upstream chat template is included as chat_template.jinja and is also stored in the GGUF metadata. The official template supports thinking and XML-style tool calls, but tool-call behavior is not certified by this release's smoke test.

Fidelity measurements

The table below compares each published GGUF with the BF16 reference on a held-out WikiText pilot: 16 chunks from wiki.test.raw and 16 chunks from wiki.valid.raw, with a 4,096-token context and the same llama.cpp runtime. Values are averaged across the two splits. Lower Mean KLD, ΔPPL, and RMS Δp, and higher Top-1 agreement, indicate closer next-token behavior to BF16. The BF16 reference mean PPL was 11.7502 in this pilot.

File	Mean KLD ↓	Top-1 vs BF16 ↑	ΔPPL	RMS Δp
MiniCPM5-2B-Q8_0.gguf	0.001514	97.792%	+0.133%	0.999%
MiniCPM5-2B-Q6_K.gguf	0.006022	95.828%	+0.140%	1.933%
MiniCPM5-2B-Q5_K_M.gguf	0.018555	93.028%	+1.247%	3.430%
MiniCPM5-2B-Q5_1.gguf	0.018927	92.770%	+1.003%	3.460%
MiniCPM5-2B-Q5_K_S.gguf	0.021076	92.497%	+1.301%	3.640%
MiniCPM5-2B-Q4_K_M.gguf	0.063265	87.537%	+3.384%	6.272%
MiniCPM5-2B-IQ4_NL.gguf	0.072267	86.721%	+4.668%	6.721%
MiniCPM5-2B-IQ4_XS.gguf	0.074946	86.399%	+5.338%	6.892%
MiniCPM5-2B-Q3_K_L.gguf	0.160423	80.606%	+16.239%	10.157%
MiniCPM5-2B-Q3_K_M.gguf	0.182391	79.684%	+17.441%	10.824%
MiniCPM5-2B-IQ3_M.gguf	0.197723	78.853%	+15.540%	11.770%
MiniCPM5-2B-IQ3_S.gguf	0.229139	77.404%	+18.694%	12.638%
MiniCPM5-2B-Q2_K.gguf	0.741842	62.160%	+88.959%	22.302%
MiniCPM5-2B-Q2_K_S.gguf	1.054076	55.437%	+156.195%	26.530%
MiniCPM5-2B-IQ2_XS.gguf	2.119478	41.038%	+632.593%	37.621%
MiniCPM5-2B-Q1_0.gguf	13.416317	0.000%	+47,867,030%	55.561%

Q5_K_S is the compact modern Q5 option; Q5_1 is included as a legacy-format comparison. Q6_K remains the Q6 control in this release.

The compact machine-readable results are available in reproducibility/quality-summary.tsv, with the corpus hashes, evaluation profile, and runtime commit recorded in the reproducibility/manifest.md.

Quick start

With a recent llama.cpp build:

./llama-cli \
  -m MiniCPM5-2B-Q4_K_M.gguf \
  --chat-template-file chat_template.jinja \
  --jinja \
  --reasoning off \
  -p 'Answer briefly in English: What is GGUF and why is it useful for running language models locally?' \
  -n 128 -c 4096 -ngl 99

The upstream model card documents vLLM, SGLang, Transformers, llama.cpp, Ollama, and LM Studio deployment paths. Use a runtime version that supports MiniCPM5 and the model's chat template.

Reproducibility and validation

The reproducibility/manifest.md locks the upstream revision, source hashes, converter/runtime commit, calibration data, imatrix hash, quantization commands, validation profile, and benchmark profile. The package also includes the model-specific calibration corpus, combined imatrix, quantization script, smoke-test script, and compact quality summary.

Published files passed load/generate smoke tests with tensor checking and the official MiniCPM5 chat template. Raw conversion, quantization, imatrix, smoke-test, quality-evaluation, and benchmark logs are retained locally and are not included here.

License and attribution

The upstream model is released under the Apache License 2.0. Preserve the upstream attribution and LICENSE when redistributing these derivative files. These are community GGUF quantizations, not an OpenBMB release or endorsement.