GSQ-RCO quantized GGUF build of the stock openbmb/MiniCPM5-2B weights, made for llama.cpp.
openbmb/MiniCPM5-2B (developer: OpenBMB) is a dense 2B-class causal language model built on the standard LlamaForCausalLM architecture: 42 layers, GQA (16 query heads / 2 KV heads), 2048 hidden, 6144 intermediate, 130,560-token vocab, 131,072-token context. Total 2,516,756,480 parameters.
License: Apache-2.0 (inherited from the base model).
| File | Size | BPW | Ladder (42-layer, mixed IQ) |
|---|---|---|---|
quant/MiniCPM5-2B-GSQ-RCO-IQ3_S.gguf |
1.14 GiB | 3.88 whole-file (3.48 selected MLP) | all attn_* @ IQ4_NL; per-tensor MLP mix of IQ1_M–IQ4_NL; token_embd/output @ IQ4_NL; norms fp32 |
Only the final quantized build is hosted here. The source checkpoint is linked below instead of being mirrored in this repository.
openbmb BF16 master
→ BF16 GGUF conversion (no intermediate quantization)
→ this 3.5-tier build
Source: openbmb/MiniCPM5-2B @ revision f97400052a43d642bbc6e9975e2397e3ae6a6b52 — vendor BF16 safetensors, converted directly to BF16 GGUF for this build.
Importance signal: instead of an imatrix, the per-tensor bitwidth assignment was optimized by the GSQ-RCO pipeline, calibrated against the stock weights on the HuggingFaceFW/fineweb-edu calibration set (32 sequences × 256 tokens). Objective: cross-entropy — measured NLL 4.888 / KL 2.109 against the BF16 reference at the assigned bitwidths.
The bitwidth assignment was produced by the actual GSQ/RCO 3-bit recipe, run on the BF16 checkpoint, then emitted as a per-tensor mixed-precision GGUF using llama.cpp's IQ quant formats. This build targets the 3.5 tier.
The allocation spans every 2-D matrix: dense MLPs (ffn_gate, ffn_up, ffn_down), and attention (attn_q, attn_k, attn_v, attn_output). Fixed policies:
All attention weight matrices: IQ4_NL (42/42).
MLP matrices: per-tensor mixed ladder of IQ1_M, IQ2_XXS, IQ2_S, IQ2_XS, IQ3_XXS, IQ3_S, IQ4_XS, IQ4_NL — selected by GSQ/RCO, averaging 3.48 bpw across MLP weights.
token_embd / output: IQ4_NL (tied-vocab-size embeddings, 130,560 × 2048).
1-D norms/biases: fp32.
Whole-file size is 1.14 GiB (3.88 bits per source weight element, 2.52B params).
The build was verified as an exact tensor-payload match to the optimized assignment (381 tensors), and smoke-tested for load and coherent output with pinned llama.cpp CPU inference (short chat prompts).
llama-cli -m quant/MiniCPM5-2B-GSQ-RCO-IQ3_S.gguf \
-p "Explain mixed-precision quantization." -ngl 99
Recommended sampling from the upstream card: temperature=1.0, top_p=0.95, min_p=0.0 (llama.cpp's default min_p=0.05 can cause repetition loops with this model).
This is a single 3.5-tier IQ3_S build; no 2-bit or 3.0-tier variants are hosted here.
These builds come directly from the vendor BF16 master; no intermediate quantization step is involved.
The smoke tests used short completion/chat prompts only; this is not a benchmark accuracy result.
The bitwidth ladder was optimized against the BF16 reference on a small web-text calibration sample; no published benchmark was re-run at this precision.
openbmb/MiniCPM5-2B