MiniCPM-V-4.7-1B-W8

Weight-only int8 version of openbmb/MiniCPM-V-4.7-1B. The vision tower, projector, language model linears, lm_head and token embeddings are stored as int8 with fp16 group scales. Norms, biases, the MTP head and small tensors stay fp16.

The weights use a custom packed format and are loaded with the w8_loader.py file included in this repository. Standard from_pretrained does not load them.

In plain words

This is a smaller copy of MiniCPM-V-4.7-1B. The original model looks at pictures and answers in text.

The stored weights are about half the size: 1552 MiB instead of 2965 MiB. While generating, the GPU peak is about 43% lower than the original (1752 vs 3052 MiB allocated, with the current loader).

What was lost, as far as we could measure:

• Text: almost nothing. Perplexity (a standard score of how well the model predicts text; lower is better) changed by +0.16%.
• Picture captions: the score moved by -2.65%. A negative number here does not mean the model got better. The same test jumped between -3% and +2.3% for nearly identical settings, so it is too noisy to say whether captions got better or worse.
• Honest summary: no damage was measured. But the copy is not proven to be identical to the original. No task tests were run, such as answering questions about images or reading text in images.

It is not faster. It uses less memory for the weights, but it generates roughly 15 to 30% slower than the original (about 14.5 vs 17.9 tokens per second on a T4 in our tests). If you want speed, use the GGUF version in ewin-reg/MiniCPM-V-4.7-1B-W8-GGUF, which runs in llama.cpp and was faster than f16 in our test.

It is not a drop-in replacement. You need the included w8_loader.py, and the usual from_pretrained will not open it. Copy this:

from w8_loader import load_w8
model, missing, unexpected = load_w8("path/to/this/folder")
model = model.cuda()

Use this copy if you want the smaller download or to study the quantized weights; for running on small hardware, use the GGUF version. If you need the best accuracy, or reliable reading of text in images, use the original model.

Glossary: int8 is 8-bit integer storage for weights. Perplexity is how surprised a model is by real text; lower is better. NLL (negative log-likelihood) is the average surprise per token; perplexity is its exponential.

Model description

MiniCPM-V 4.7 is a multimodal model from OpenBMB that accepts images and text and generates text. The 1B variant is described in the base model card as supporting single-image, multi-image and video understanding. Its architecture is named minicpmv4_7 and requires transformers>=5.18. The components are:

• Vision encoder: SigLIP2 vision tower with 27 layers.
• Language model: Qwen3.5 hybrid model with 24 layers, of which 18 use linear attention and 6 use full attention.
• Output: untied lm_head and an MTP head.
• No audio.

The original bf16 checkpoint is 3.11 GB (1554.5M parameters).

This repository contains the same architecture with the weights stored in int8. The stored size of model.safetensors is 1552.0 MiB, compared with 2964.9 MiB for the fp16 weights, which is 47.7% smaller. The file contains 358 int8 tensors.

Intended uses

This is a compressed copy of the base model with smaller stored weights. It is intended for the same tasks as the base model: image-text chat and captioning. Behaviour should match the base model closely according to the text perplexity measurement below, but task-level accuracy has not been measured.

Limitations

• Not benchmarked on VQA, OCR or document tasks. No task benchmark results (VQA, OCR, MMBench or similar) were run for this quantized model, so benchmark accuracy is unmeasured.
• Custom loader, not a standard quantization format. It works with the included w8_loader.py, not with generic loaders for GPTQ, AWQ or GGUF.
• Evaluated only on English captions and wikitext.
• Inherits the biases and risks of the base model (see below).

How to get started

Requirements: transformers>=5.18, safetensors, torch. Place w8_loader.py next to your script or on the Python path.

from transformers import AutoProcessor
from w8_loader import load_w8

path = "path/to/this/folder"
processor = AutoProcessor.from_pretrained(path)
model, missing, unexpected = load_w8(path)
model = model.cuda()

load_w8 returns the model together with the lists of missing and unexpected keys. The loader builds the model skeleton from config.json with empty weights, then replaces each quantized linear and embedding with a module that keeps int8 weights in memory and dequantizes one layer at a time in fp16 during the forward pass (large layers such as lm_head in pieces of 64 MiB). Optionally set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before starting Python to lower reserved GPU memory.

Use the processor and chat template from this folder or from the base model. For images, pass downsample_mode as in the base model card: "16x" by default, or "4x" for finer detail. It must be passed to both apply_chat_template and generate.

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"},
            {"type": "text", "text": "What causes this phenomenon?"},
        ],
    }
]

downsample_mode = "16x"

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_slice_nums=36,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(
    **inputs,
    downsample_mode=downsample_mode,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.95,
    top_k=100,
    min_p=0.0,
    presence_penalty=0,
    repetition_penalty=1.03,
)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

The example and sampling parameters above follow the base model card. They were not all run against this packed model: the checks for this repository used greedy decoding with downsample_mode="16x" and max_slice_nums=9. Use fp16 on a T4 GPU, which has no native bf16 support.

Because dequantization happens per layer at runtime and there is no fused kernel, this format saves memory but is not faster than fp16 inference (see Measured memory and speed).

Quantization method

The model was produced by post-training quantization only, run on a Kaggle T4 GPU. No training, fine-tuning or distillation was performed.

Calibration used 128 samples, consisting of 48 image-caption prompts from Flickr30k and 80 wikitext-2 paragraphs. GPTQ was used for the vision tower, projector and lm_head with forward passes over the calibration samples. For the language-model linears, GPTQ at group size 64 was tried first and scored worse on held-out captions, so round-to-nearest at group size 32 was chosen. Calibration data is disjoint from the evaluation data.

4-bit weights were also tested and rejected. GPTQ at 4 bits on the large text modules (MLP, linear-attention out_proj, in_proj_qkv and in_proj_z) was far worse than the 1% limit used for this work.

Sensitivity checks showed that the vision tower, projector and lm_head each changed caption NLL by about 0%. The loss came from the text linears.

Measured memory and speed

Measured on a Kaggle T4 GPU: batch size 1, one image with a 212-token prompt, 128 new tokens with greedy decoding, best of 3 runs. Each row ran in its own process. "Old loader" is the first version of w8_loader.py (still available as an earlier revision of this repository). "Current loader" is the version in this repository now. The weights are the same in both; only the loader changed. The two loaders produced identical output: all 128 generated token ids matched.

What this means:

• With the current loader the GPU peak during generation is 1752 MiB allocated, which is 43% below the original (3052 MiB), and 1788 to 1894 MiB reserved. The old loader expanded each layer to fp16 with two full-size temporary copies, which made lm_head alone need about 1 GB at once. The current loader expands it in 64 MiB pieces.
• Loading takes about 1.4 s instead of 78 s, and CPU RAM while loading is about a third lower, because the model is first created with empty weights instead of random ones.
• Decoding is still slower than fp16: about 14.5 tokens/s against 17.9 in our runs. The old loader measured 12.57 tokens/s in an earlier run and 14.82 in this one, so speed varies by about 15% between runs; treat "roughly 15 to 30% slower" as the honest range.
• The fp16 column came from an earlier run of the same kind of Kaggle T4 kernel, not the same process as the other three columns.
• This is one run on one GPU with one prompt. Numbers on other hardware or longer prompts will differ.
• To lower the reserved GPU memory a little more, set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before starting Python.

Lower GPU memory with offload=True

If the GPU is the scarce resource, call load_w8(path, offload=True) (then .cuda() as before). The vision tower and the token-embedding table stay in CPU memory and are copied to the GPU only while they run. It is off by default because it moves memory rather than removing it. Same Kaggle T4 test, same prompt, 128 identical output tokens:

Use it when GPU memory is the limit and CPU RAM is plentiful. Each image costs extra time because the vision weights are copied over PCIe every time an image is encoded; many images or video frames will make that cost larger than in this single-image test, which was not measured.

• For speed, use the GGUF version: ewin-reg/MiniCPM-V-4.7-1B-W8-GGUF. On the same GPU its text generation ran about 1.65 times faster than an f16 GGUF.

Evaluation

Evaluation compares the reloaded packed model against the fp16 baseline, using teacher forcing on held-out data: 64 wikitext paragraphs and 128 Flickr30k images (lmms-lab/flickr30k) with their reference captions. Values are changes relative to fp16.

Final model (RTN g32 for language-model linears, int8 embeddings):

Candidate settings compared (vision, projector and lm_head use GPTQ g64 in every row):

How to read these numbers:

• Text perplexity is the more reliable measure. The chosen setting changed it by +0.158%.
• Caption NLL is not reliable enough to prove the model is lossless on images. Between near-identical int8 settings it moved by about plus or minus 2.5%, in both directions (+2.3% for RTN g64 and -2.7% for RTN g32). That spread is larger than a 1% threshold. The negative caption numbers should not be read as an improvement.
• Earlier fake-quant runs on 48 images showed 98 to 99% top-1 next-token agreement with fp16 for int8 variants. This is a separate and smaller check.

Bias, risks and safety

This model inherits the biases, risks and safety properties of openbmb/MiniCPM-V-4.7-1B and of its language and vision backbones. Quantization was not evaluated for changes in bias, toxicity or harmful output, and no such evaluation was performed for this repository. Outputs should be checked before use, particularly in high-stakes settings.

Environmental impact

Quantization was carried out in a short run on a single Kaggle T4 GPU. No training was performed.

Citation and acknowledgements

This model is a quantized derivative of MiniCPM-V 4.7 by OpenBMB. Please cite the base model; see the base model card for its citation details and acknowledgements. The base card acknowledges Qwen3.5 for the language backbone, SigLIP2 for the vision module and Transformers.

Quantized and packaged by ewin-reg.