MiniCPM-V-4.7-1B-W8-GGUF

GGUF files of openbmb/MiniCPM-V-4.7-1B in Q8_0 (8-bit), for llama.cpp. Unlike the safetensors repo ewin-reg/MiniCPM-V-4.7-1B-W8, these load with standard llama.cpp tools and need no custom loader.

In plain words

This is the same model as the original, squeezed to about half the size in the text part, so the download is smaller and it generates faster in llama.cpp. You give it a picture and a question, and it answers in text.

• The text part went from 1931 MiB to 1031 MiB.
• On the text test we ran, the quality change was well under 1% (0.24% and 0.37%).
• It generates text about 1.65 times faster than the uncompressed GGUF on a T4 GPU.
• We did not test how well it answers questions about images or reads text in images. If you need the best accuracy, use the original model.
• Words used here: Q8_0 is an 8-bit weight format used by llama.cpp. Perplexity is a text-prediction score; lower is better. GGUF is the single-file model format llama.cpp loads. mmproj is the separate file holding the vision part, needed to give the model images.

Model description

This is openbmb/MiniCPM-V-4.7-1B, a multimodal model that takes images and text as input and produces text. It has a SigLIP2 vision tower with 27 layers and a Qwen3.5 hybrid language model with 24 layers, of which 18 use linear attention and 6 use full attention. It has an untied lm_head and an MTP head, and no audio input. The base model has 1554.5M parameters and uses architecture name minicpmv4_7. License: apache-2.0.

Intended uses

Same tasks as the base model: image-based chat and image captioning, with text answers. These files are meant for llama.cpp and other GGUF runtimes built on it. Only llama.cpp was used for the checks in this card.

Files

Quantization method

How they were made:

• Post-training quantization only. No training, fine-tuning or distillation.
• The W8 recipe in the safetensors repo: text linears int8 with round-to-nearest, group size 32; lm_head and vision weights GPTQ 8-bit, group size 64, with 128 calibration samples. The full recipe and its evaluation are in ewin-reg/MiniCPM-V-4.7-1B-W8: text perplexity +0.158%; caption NLL -2.654%, which moved too much between runs to read as an improvement.
• The W8 text file starts from the int8 weights of ewin-reg/MiniCPM-V-4.7-1B-W8. Those were dequantized back to fp16, converted to GGUF, and quantized to Q8_0 with llama-quantize. Text linears there use group size 32, which matches the Q8_0 block size, so those weights should map back to the same int8 codes. The lm_head, embeddings and vision weights used group size 64 in the W8 model and are only close, not bit-identical, after re-quantizing.
• The plain text file is the original weights converted to f16 GGUF and quantized with llama-quantize to Q8_0, with no step through our W8 weights.
• The vision file was converted with convert_hf_to_gguf.py --mmproj --outtype q8_0 from the dequantized W8 weights. By llama.cpp's rules, norm weights and patch embeddings stay f32 and the other 2D weights become Q8_0.

How to use

Requires a llama.cpp build recent enough to include MiniCPMV4_7 support (current master).

llama-mtmd-cli -m MiniCPM-V-4.7-1B-W8-Q8_0.gguf --mmproj mmproj-MiniCPM-V-4.7-1B-Q8_0.gguf --image picture.png -p "Describe this image." -c 4096 -ngl 99

The model may print a thinking block before the answer, as the base model does.

Measurements

All numbers come from one run on a Kaggle T4 GPU with llama.cpp built from master (CUDA, architecture 75). Perplexity used llama-perplexity with context 512 on 64 paragraphs (over 200 characters each) from the wikitext-2 test split. Speed used llama-bench with 512 prompt tokens and 128 generated tokens, 3 repeats, all layers on the GPU.

Notes on reading these numbers:

• Perplexity is on a different text sample and tool than the safetensors repo, so the percentages are not directly comparable to the +0.158% reported there.
• The gap between the plain and W8-derived files (0.13 percentage points) is small and was measured once, so do not treat it as a reliable ranking.
• Prompt processing is 7 to 15% slower than f16 here; text generation is faster.
• Image quality was only checked by one caption, which was coherent. No image caption loss, VQA, OCR or other task benchmark was run for these GGUF files. The vision tower is quantized in addition to the text model, which the safetensors W8 repo also does.

Limitations and risks

• Accuracy on image tasks, document reading and OCR is unmeasured.
• Evaluated on English wikitext only.

Bias, risks and safety

Inherits the biases and risks of openbmb/MiniCPM-V-4.7-1B. Quantization was not evaluated for changes in bias or harmful output.

Environmental impact

Only one short run on a Kaggle T4 GPU was used for the measurements above. No training was done, and no energy or emissions figures were measured.

Citation

No BibTeX citation is available for this repo. Cite the base model as described in the base model card.

Acknowledgements

Quantized derivative of MiniCPM-V 4.7 by OpenBMB; see the base model card for citation details. Conversion and quantization used llama.cpp. Packaged by ewin-reg.