MiniCPM-V-4.7-35B-A3B-NVFP4-GGUF

HF repo: esatapedico/MiniCPM-V-4.7-35B-A3B-NVFP4-GGUF (this family). The NVFP4 safetensors checkpoint lives in its own repo, esatapedico/MiniCPM-V-4.7-35B-A3B-NVFP4, with its own card.

A family of six GGUF files of openbmb/MiniCPM-V-4.7-35B-A3B (Apache-2.0): a 35B mixture-of-experts vision-language model (40 layers, 256 routed experts per layer + shared expert, hybrid linear-attention + full-attention layers, SigLIP2-style vision tower). Different from everything else on this account: the NVFP4 here is our own conversion. The BF16 source was quantized with llm-compressor instead of starting from a third-party NVFP4 checkpoint, then converted to GGUF with llama.cpp and built into a size/precision ladder for the tensors that most affect output quality and decode speed.

NVFP4 here means W4A16 weight-only, group size 16: weights packed as 4-bit with FP8 per-tensor scales, activations computed in BF16.

These files need a llama.cpp build with MiniCPM-V 4.7 support (merged upstream 2026-10-10, PR #29416) plus the NVFP4 CUDA kernels for Blackwell (sm_120). Mainline is enough, no fork needed. On anything older the arch is simply unknown and the GGUF won't load.

All tiers are 18–20 GiB of weights, sized for two 16 GB cards (layer split) or partial offload on one card; a single 16 GB GPU cannot hold a full tier. See Hardware notes below.

Follow along & support

I post updates on new conversions, benchmarks, and what I'm working on over on my site. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, you will find the options there. I do this on consumer hardware and like seeing how far it goes. More is on the way.

🌐 esatapedico.com. Updates, work-in-progress, and ways to support.

The six files

Six tiers share a byte-identical 400-tensor native NVFP4 backbone (240 fused expert weights from our NVFP4 conversion + 160 attention/SSM linear projections promoted to NVFP4 at tier build), differing only in the lm_head (output.weight) / token_embd precision:

For vision input, pair any tier with mmproj-MiniCPM-V-4.7-35B-A3B.gguf (1.12 GB, vision encoder + merger, 460 tensors). Sizes are decimal GB (1 GB = 1,000,000,000 bytes), matching what the Hugging Face file browser shows.

Tensor layout

One tier (MEDIUM, Q8_0/Q6_K heads) as an example. Every tier has the same 973-tensor skeleton:

The 240 expert tensors came out of the source checkpoint already NVFP4 (see How this was made) and pass through backwards and forwards untouched (per-tensor SHA-256 verified). The 160 attention/SSM projections are BF16 in the source; the tier builds promote them to NVFP4 uniformly, which is why the backbone is identical across tiers and why there is no "HIGHEST" tier: requantizing NVFP4 is disabled by design, so the attention projections do not get a BF16-express rung like some earlier families on this account.

Why the head ladder

Same principle as the previous families: the NVFP4 backbone is already at its own noise floor (per-tensor scaling compensates the narrow dynamic range), so the dominant remaining quality lever is the vocabulary interface (lm_head + token embedding), for both quality and decode speed (see the speed table: decode tracks head precision).

DSpark speculative decoding

MiniCPM-V-4.7 ships with a DSpark drafter: a small 5-layer, 389M-parameter draft model trained by OpenBMB to pair with MiniCPM-V-4.7-35B-A3B and its tokenizer, proposing up to 15 tokens per forward pass. It is published upstream as a separate model and works with any of these tiers. It is not in this repo, and upstream files are not duplicated here.

Upstream companions (Apache-2.0):

• BF16 checkpoint: openbmb/MiniCPM-V-4.7-35B-A3B-DSpark
• GGUF drafts (recommended Q8_0 / Q6_K; a 789 MB F16 draft also ships in openbmb/MiniCPM-V-4.7-35B-A3B-gguf): openbmb/MiniCPM-V-4.7-35B-A3B-DSpark-gguf

Upstream example usage with our kind of files (-md = draft, -ngld offloads the draft):

llama-server \
  -m MiniCPM-V-4.7-35B-A3B-NVFP4-MEDIUM.gguf \
  --mmproj mmproj-MiniCPM-V-4.7-35B-A3B.gguf \
  -md dspark-MiniCPM-V-4_7-35B-A3B-Q8_0.gguf \
  --spec-type draft-dspark --spec-draft-n-max 5 \
  -ngl 99 -ngld 99 -fa on -c 8192 --jinja

Measured throughput on the MEDIUM parent (greedy, n_predict = 256, short chat prompts, -ngl 99 -ngld 99 -c 4096, f16 KV; warm-up excluded, median of 3 repetitions per row). Two independent runs agree: the first back to back on a busy box, the second in a dedicated GPU window with every other GPU workload scaled to zero. Greedy acceptance counts are byte-identical across the two runs.

Read: the drafter accepts a mean block of about 3 tokens, so at the default 15-token budget most proposals are pure overhead and speculative decoding runs about 25% slower than plain decode. Capping the budget at 5 removes enough of the waste that the upstream Q8_0 drafter is a small net win (about +5% vs no drafter). That capped configuration is the one recommended here (--spec-draft-n-max 5). The base model has no MTP head and none of these files embed one.

A self-converted draft (NVFP4A16 trunk, Q8_0 heads, 304 MB) measured 77.2 tok/s at budget 15 and 109.6 at budget 5 in the same harness (within about 2% of the upstream Q8_0 draft at 0.72x the size, with effectively tied acceptance). It is not in this repo; it is published with its own card at esatapedico/MiniCPM-V-4.7-35B-A3B-DSpark-NVFP4A16 (dspark-MiniCPM-V-4.7-35B-A3B-NVFP4A16-q8heads.gguf, 304 MB, with the measured numbers and the disclaimer).

Vision (mmproj)

The multimodal projector is included in this repo: mmproj-MiniCPM-V-4.7-35B-A3B.gguf (1.12 GB), converted from the same BF16 source at the same llama.cpp pin as the tiers, using the default 16× visual-token compression. Quick single-prompt smoke test (image + "what colour…" → correct answer) passed on the ORIG parent with this projector. That is a one-prompt check, not a vision benchmark. Known minor upstream issues to watch when serving image requests: #30508 (eval bug) and #30510 (image-prompt marker positions).

If you want 4× more visual tokens for extra detail, upstream publishes a -4x variant of the projector (openbmb/MiniCPM-V-4.7-35B-A3B-gguf); the loader picks the graph from the projector metadata, so the same tiers serve both.

Quality (measured, gate tier)

llama-perplexity --ppl on the wikitext payload, and MMLU over the /models harness (n=1000, mode=limit), all at f16 KV cache, --flash-attn on:

Every tier clears the family gate (MMLU ≥ 80.41%). Two things worth noticing:

• The NVFP4-attention promotion (tiers vs ORIG) costs nothing measurable at this scale. MMLU stays in the same band, PPL moves ~3% and not in one direction.
• Head rungs are within noise of each other (80.9–82.3%); the ladder exists mainly for size and decode speed, not for a quality gradient.

KLD-vs-anchor is N/A for this family (no BF16 anchor logits published yet), so that gate is informational only; perplexity and MMLU above are the scored gates.

Speed (measured)

llama-bench, dual Blackwell cards (RTX 5070 Ti + 5060 Ti, 16 GB each, layer split across both), -fa on, f16 KV, single run per cell. Treat as relative and hardware-specific. Do not compare against other-hardware numbers.

Read of the table: prefill is ~7.4k tok/s across the family at 2048-token prompts and mostly flat across tiers; decode tracks head precision exactly (129 t/s with Q3_K/Q2_K heads down to ~104.5 t/s with BF16 lm_head) while the NVFP4 attention promotions cost nothing measurable between ORIG and the tiers' faster path. Single-token TTFT was 133–185 ms depending on tier; that probe is far too small to mean much beyond "chat-shaped requests answer immediately".

Usage

llama.cpp / llama-server

llama-server \
  -m MiniCPM-V-4.7-35B-A3B-NVFP4-MEDIUM.gguf \
  --mmproj mmproj-MiniCPM-V-4.7-35B-A3B.gguf \
  -ngl 99 -fa on -c 8192 --jinja --port 8080

• On two 16 GB cards, this loads with a layer split automatically. On one card, use -ngl <n> to cap GPU layers, or --n-cpu-moe <n> to keep expert weights on CPU (the 3B active parameters make MoE-offload comparatively cheap; offload numbers are not measured for these tiers).
• Images (llama-cli/llama-server): pass --mmproj as above; upstream docs use llama-mtmd-cli for single turns:

llama-mtmd-cli \
  -m MiniCPM-V-4.7-35B-A3B-NVFP4-HIGH.gguf \
  --mmproj mmproj-MiniCPM-V-4.7-35B-A3B.gguf \
  -c 4096 --jinja --image photo.jpg -p "What is in the image?"

• The chat template enables thinking by default. Turn it off with --chat-template-kwargs '{"enable_thinking": false}'.
• Download the DSpark drafter separately for speculative decoding (see the section above).

How this was made

1. Our own NVFP4 checkpoint. The BF16 source (openbmb/MiniCPM-V-4.7-35B-A3B, 70.4 GB, 1,152 tensors, no MTP tensors; the DSpark drafter is intentionally excluded here) was quantized with llm-compressor (datafree, NVFP4A16 W4A16, group size 16) targeting the fused routed experts + shared expert only. Every norm tensor was verified byte-identical to the source after quantization (a known upstream norm-perturbation issue was guarded against). This checkpoint is published in the sibling repo.
2. GGUF conversion. llama.cpp @ 69f201a (convert_hf_to_gguf.py --outtype auto keeps the NVFP4 tensors as native GGML type 40), yielding the ORIG parent (22.89 GB; 240 NVFP4 + 192 BF16 + 541 F32).
3. Tier ladder. Each tier built with llama-quantize --tensor-type-file <overrides> from the ORIG parent: the 160 attention/SSM projections promoted to NVFP4 identically in every file, plus the per-tier head quantization from the table. Backbone identity across tiers verified per-tensor (400/400 identical; the 240 expert tensors also byte-identical to the source-based parent).
4. mmproj converted with convert_hf_to_gguf.py --mmproj from the same source at the same pin.

Per-tier override maps ship in this repo (overrides-* files) for reproduction.

Attribution & provenance

Derivative work built from Apache-2.0 artifacts. Nothing here was trained. Credit belongs to:

1. OpenBMB for the base model openbmb/MiniCPM-V-4.7-35B-A3B, the DSpark drafter, and the official GGUF companions.
2. llama.cpp contributors for the GGUF toolchain, the NVFP4 (GGML type 40) path with Blackwell kernels, and the MiniCPM-V 4.7 conversion/serving support (PR #29416).
3. Red Hat AI / IBM for llm-compressor (via vLLM), used for our NVFP4 quantization.
4. esatapedico for the NVFP4 conversion, the GGUF conversion, tier ladder, mmproj conversion, verification, and benchmarking.

"MiniCPM-V" is a name of OpenBMB. This repository is not affiliated with, sponsored by, or endorsed by OpenBMB or Red Hat AI.

Repository contents

• MiniCPM-V-4.7-35B-A3B-NVFP4-VERY-LOW.gguf (19.40 GB)
• MiniCPM-V-4.7-35B-A3B-NVFP4-LOW.gguf (19.64 GB)
• MiniCPM-V-4.7-35B-A3B-NVFP4-MEDIUM.gguf (19.98 GB)
• MiniCPM-V-4.7-35B-A3B-NVFP4-MID-HIGH.gguf (20.10 GB)
• MiniCPM-V-4.7-35B-A3B-NVFP4-HIGH.gguf (20.45 GB)
• MiniCPM-V-4.7-35B-A3B-NVFP4-VERY-HIGH.gguf (21.05 GB)
• mmproj-MiniCPM-V-4.7-35B-A3B.gguf (1.12 GB)

SHA-256

e1a6d954a307680dfae7e52bf1e95913bb5871752c91e5b0c26496ec548afcc6  MiniCPM-V-4.7-35B-A3B-NVFP4-VERY-LOW.gguf
65a44e4a1d13515936b9f77f83fe677468b090297bad62db907b6cacdb34cb48  MiniCPM-V-4.7-35B-A3B-NVFP4-LOW.gguf
d00636178e6293f4f6515694c1210230aa95c83ebb02433bec802972b9a953aa  MiniCPM-V-4.7-35B-A3B-NVFP4-MEDIUM.gguf
0354adc81e9637edefd2d7c45ecce6efaeb46104af25174bae0d1662353350ea  MiniCPM-V-4.7-35B-A3B-NVFP4-MID-HIGH.gguf
759b989ebd1de97d14f05548c89d633df82d2db980280262e98d47dec605a752  MiniCPM-V-4.7-35B-A3B-NVFP4-HIGH.gguf
6b0c8eeb20920f43c9ef3cdc4505a6e9ef034e8d409d3b963e2af0aee2c04e1c  MiniCPM-V-4.7-35B-A3B-NVFP4-VERY-HIGH.gguf
cf2bfabfeb6cc9af830b3bbbbfc1c71cbe392a3d445f50c13e38902f119a0b4c  mmproj-MiniCPM-V-4.7-35B-A3B.gguf

Hardware notes

• Tested on 2× NVIDIA Blackwell 16 GB (RTX 5070 Ti + RTX 5060 Ti), layer split. All tiers fit this shape.
• A single 16 GB card cannot hold a full tier (weights alone are 18–19.6 GiB). Options that keep the NVFP4 backbone: dual-card layer split, --n-cpu-moe expert offload, or partial -ngl offload with the smaller tiers. There is no "16 GB-fit" file in this family on purpose: the NVFP4 expert backbone is already the size floor, and re-quantizing NVFP4 elsewhere would break what makes these files what they are.
• Blackwell (sm_120) required for native NVFP4; this is a NVFP4-first family, no K-quant variants ship here.

License

Apache-2.0, identical to every upstream artifact. The base model license governs; GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to OpenBMB (base model) as above.

Note on this card

This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the file itself before relying on it.