MiniCPM-V-4.7-35B-A3B-INT4-W4A16-LowLatency

A latency-first INT4 (W4A16) build of openbmb/MiniCPM-V-4.7-35B-A3B for real-time, single-stream multimodal chat on vLLM. Routed experts, Gated-DeltaNet and attention are all INT4; lm_head is INT8; the vision tower is untouched. The AWQ router bug found in the official build is fixed. The result: 4.57 ms per token, about 1.55× faster decode than every official quantized build.

Why this build. OpenBMB's official int4 / AWQ / GPTQ / FP8 checkpoints are made the usual way: quantize the routed experts (FP8 also the attention projections) and keep the rest in BF16. That is a sound default for throughput. This build targets the other regime: a voice agent or a robot head, where one GPU turns a ~1.5 k-token persona prompt (plus, now and then, a camera frame) into a 20- to 60-token spoken reply, thinking off, one stream at a time.

On a 3 B-active MoE, the BF16 parts the official builds keep are most of the bytes every decode step reads, so we quantize them too. The price is measured and shown below: about 1.5 points less top-1 agreement with BF16 than the official GPTQ, with C-Eval / MMLU / GSM8K within 0.6 points.

TL;DR

• Size: 21.8 GB (BF16 base 70.6 GB).
• Format: INT4 W4A16 compressed-tensors on vLLM's Marlin kernels; vision tower BF16 (byte-identical to the base), lm_head INT8.
• Serving: vLLM 0.32.0, no --trust-remote-code.
• Decode: 4.57 ms/token, vs 7.05–7.30 for the official int4 / AWQ / GPTQ / FP8.
• Per reply: a ~30-token reply after a 1.5 k-token prompt takes ≈ 243 ms, vs ≈ 330–341 ms (−26 %).
• First token: TTFT 80 ms; 141 ms with a 640×480 camera frame.
• Benchmarks: C-Eval / MMLU / GSM8K 84.3 / 86.8 / 95.4 (BF16 84.5 / 87.3 / 95.8).
• Fidelity: top-1 vs BF16 91.4 % (official GPTQ 92.9 %, official AWQ 85.5 %).
• Images: 48/48 on a synthetic image check (BF16 47/48).

All numbers: one RTX 6000 Ada, batch 1, the same harness for every build.

Quantizing GDN and attention as well is what made our Qwen3.6-35B-A3B LowLatency build ~40 % lower in ms/token on the same backbone. Here it gives −36 % vs the official GPTQ.

Quick start (vLLM ≥ 0.32.0)

Recipe A: voice / chat with an occasional camera frame (recommended)

vllm serve Twu31/MiniCPM-V-4.7-35B-A3B-INT4-W4A16-LowLatency \
  --served-model-name minicpmv47 --max-model-len 16384 --max-num-seqs 8 \
  --enable-prefix-caching --default-chat-template-kwargs '{"enable_thinking": false}'

On one RTX 6000 Ada:

• Text turn: TTFT 80 ms, 4.57 ms/token, ≈ 243 ms per ~30-token reply after a 1.5 k-token prompt.
• With a 640×480 frame: TTFT 141 ms. The frame adds ~213 prompt tokens in the default 16x mode.

Recipe B: text only Same command, plus --language-model-only. This skips the vision tower (1.1 GB of weights).

import base64
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
frame = base64.b64encode(open("frame.jpg", "rb").read()).decode()
r = client.chat.completions.create(model="minicpmv47", max_tokens=64, temperature=0.7,
    messages=[{"role": "system", "content": "你是一个简洁、友好的机器人助手。"},
              {"role": "user", "content": [
                  {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{frame}"}},
                  {"type": "text", "text": "我手里拿的是什么？"}]}])
print(r.choices[0].message.content)

• Kernels: no --quantization flag is needed. vLLM logs Using MarlinLinearKernel for CompressedTensorsWNA16.

• vLLM version: 0.32.0 is the first release with MiniCPM-V 4.7. It needs Python ≥ 3.11 and transformers 5.18–5.19. No --trust-remote-code is needed: the processor, tokenizer and chat-template files are byte-identical to the base repo's transformers-native ones.

• Memory: the weights take 20.3 GiB in vLLM. In the same harness the official int4 / AWQ / GPTQ take 21.1 GiB and FP8 35.5 GiB.

• DSpark draft: for short conversational replies it made decoding slower on every build we tried. On 32 Chinese chat turns, acceptance was 1.8 tokens/step:

  • official GPTQ: 9.9 ms/token at k = 15, 8.2 at k = 7, vs 7.8 without
  • this build: 6.4 at k = 7, vs 5.25 without

  Use it for long math / code outputs only, and measure.

• Thinking: the chat template enables thinking by default. Every number on this card is with it off.

Latency

RTX 6000 Ada 48 GB, vLLM 0.32.0, CUDA graphs, prefix caching off, batch 1. Prompt: a ~1.5 k-token Chinese persona system prompt + 4 history turns + the user turn. "Fixed" means 1-token and 257-token requests with ignore_eos; "natural" means 32 different user turns, greedy, natural EOS (replies ~30 tokens). Medians. The "~30-token reply" column is derived from those medians, as natural TTFT + 30 × natural TPOT.

Fidelity

Stage-1 harness (same as our Qwen3.6 cards). Every build scores identical token ids with vLLM prompt_logprobs (top-20), eager mode, against BF16 MiniCPM-V-4.7 (CPU-offloaded on the same card). The data: 100 zh-Wikipedia + 100 en-wikitext-2 chunks × 992 scored tokens, plus 450 synthetic Chinese chat turns answered by BF16 (16,471 scored answer tokens). KL = truncated KL(BF16 ‖ build) over BF16's top-20.

This build's routed experts use finer groups than the official ones (g32 vs g128). Most of its gap to the official GPTQ is therefore likely the price of also quantizing GDN and attention, which is what buys the speed. We have not split that gap per module on this model.

What the tensors show. We audited openbmb/MiniCPM-V-4.7-35B-A3B-AWQ@e61c9b9 and -GPTQ@21548bb against BF16 on 2026-10-19.

• Routers in the official AWQ were not compensated. All 40 post_attention_layernorm weights were divided by AWQ scales, but all 40 MoE routers (mlp.gate), which read the output of those norms, are byte-identical to BF16. Its recipe passes an explicit AWQ mapping list whose post-attention entry has the experts, the shared expert and shared_expert_gate, but not the router.
• The official AWQ and int4 differ only by AWQ smoothing. The AWQ build's quantization step is the same as the official -int4's (W4A16 g128 round-to-nearest, same ignore list). -int4 scores 93.13 % / KL 0.031, -AWQ 85.50 % / 0.161. We have not measured separately how much of that gap comes from the routers. Reported upstream: llm-compressor#3337 (fix in #3338).
• Norm round trip. 101 of 131 norms in the official GPTQ (and 61 in the AWQ) equal bf16(bf16(1 + w) − 1) instead of w. This is llm-compressor 0.14's offset-norm round trip (issue #3277).

What this build does about it:

• It adds the router to that AWQ mapping. A random per-channel scale test leaves the logits unchanged to ~1e-6, vs ~0.7 relative change without the router.
• It restores every norm it did not smooth, byte-for-byte.
• In layers 2–5, AWQ's smoothed (1 + w) is too small to be stored accurately in bf16. There it adjusts the scales so that the stored norm and the layers compensating it stay exactly consistent. The effect is small here (+0.2 points top-1 vs without).

Images

A sanity check, not a benchmark. It uses synthetic 640×480 images generated from a fixed seed:

• 48 images with known answers, 8 each: Chinese text, a Latin/number code, counting circles, color, left/right, and the tallest bar in a chart. Chinese prompts, greedy, thinking off.
• 24 scenes (shapes + a word + a small bar chart) described by BF16. Every build then scores BF16's descriptions teacher-forced (3,070 tokens), with the same top-1 / KL definition as above.

The only differing answer is one code that BF16 and the official GPTQ read as SMKHD instead of SMKKHD; this build reads it correctly. On image-conditioned text, the gap to the official GPTQ is about the same as on plain text above.

Benchmarks

C-Eval val (1,346), MMLU (20 per subject, 1,140), GSM8K (500); greedy, thinking off, no system prompt. Multiple choice is scored on the A/B/C/D logits after the prompt. RTX 6000 Ada, vLLM 0.32.0 (BF16 with 34 GiB CPU offload). Same items and harness as our Qwen3.6 cards. One item ≈ 0.07 / 0.09 / 0.2 points.

A plain-use difference: asked "你好，请用一句话介绍你自己。" with thinking off and greedy decoding, the official AWQ build answers that it is an assistant developed by OpenAI. The official FP8, int4 and GPTQ builds answer MiniCPM / OpenBMB.

How it was made

1. Text master. The LLM (model.language_model.* + lm_head) was re-saved as a plain Qwen3_5MoeForCausalLM, with bit-exact logits vs the MiniCPM class. The vision tower and merger never pass through the quantizer.
2. PTQ (llm-compressor 0.14.0): AWQ with the router added to the post-attention mapping and bf16-storable norm scales, then GPTQ.
   • Routed experts: INT4 g32. GDN + attention: INT4 g64.
   • Shared experts, router, GDN a/b, norms, embeddings: BF16.
   • lm_head: INT8 g128 RTN.
   • Calibration: 256 × 2048 tokens of public instructions (alpaca-zh, alpaca), answered through the chat template by the official MiniCPM-V-4.7 FP8 build. Thinking off; half of them carry a short system prompt.
3. QAT-LoRA: tried, not released. We trained a LoRA through the INT4 grid (GDN / attention) and on the shared experts, distilled from BF16's top-20 on 5,221 public conversations. It did not lower held-out KL (0.0340 → 0.0344) or improve the Stage-1 numbers above, so the released weights are the PTQ build.
4. Audit before upload:
   • vision tensors byte-identical to the base repo
   • no round-trip norm drift
   • routers compensated
   • (1 + w)·W unchanged (to bf16 rounding) for every BF16 layer next to a smoothed norm
   • every BF16 linear covered by the ignore list

No private or user data was used anywhere in this build.

Limitations

• No MTP head (the base checkpoint ships none), so no MTP speculative decoding.
• Measured with thinking off; long-form thinking outputs were not evaluated.
• Calibration data is text-only. Images were checked only on the small synthetic set above, with no natural-image benchmark. The vision tower itself is byte-identical to BF16.
• Fidelity is 1.5–1.7 points top-1 below the official GPTQ / int4. If you are throughput- or quality-bound rather than latency-bound, those are the better choice.
• Latency numbers are batch 1; at high concurrency the advantage shrinks.

Reproducibility

• Base: openbmb/MiniCPM-V-4.7-35B-A3B@0bbda78.
• Official builds compared: -int4@1fe0fee, -AWQ@e61c9b9, -GPTQ@21548bb, -FP8@25f6715, and the DSpark draft -DSpark@55249d3.
• Quantization: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.13.0.
• Serving and measurement: vLLM 0.32.0, transformers 5.19.0.
• Hardware: one RTX 6000 Ada 48 GB, for both quantization and measurement.