MiniCPM-V 4.7 35B-A3B — SAGE EXL3 3.74 bpw

Run it: DGX Spark recipe · RTX 4090 recipe. Both serve the full 262,144-token context through TabbyAPI.

An EXL3 quantization of OpenBMB MiniCPM-V 4.7 35B-A3B, released under the SAGE name. This is a standalone quantized checkpoint; the original weights are not needed at load time. The upstream model card declares Apache-2.0; this card keeps that license attribution and links the exact upstream base.

The shipping weights are a tensor-identical compact re-shard of the selected quantization. All 125,215 tensors were checked twice against the source pack by key, dtype, shape and raw-byte SHA-256 before release.

At a glance

Vision precision

The vision path is stored at 8-bit precision. This is not a full-precision vision checkpoint. On the exact shipping files, a native ExLlamaV3 smoke test produced distinct, finite 64×2048 FP16 embeddings for synthetic red and blue images and returned the correct one-word color for both under greedy decoding. This is a functional validation, not a replacement for the upstream multimodal benchmark suite.

Load and serve

Install a compatible CUDA-enabled PyTorch and ExLlamaV3 build first. From an ExLlamaV3 checkout, the upstream interactive example takes a local model directory and a prompt format:

python examples/chat.py -m /path/to/model-directory -mode <prompt-mode>

For an OpenAI-compatible API, TabbyAPI is the ExLlamaV3 backend. Put the downloaded model folder under its configured model_dir, set model.model_name in config.yml to that folder name, then launch:

python main.py

The API defaults to http://127.0.0.1:5000; query GET /v1/models to confirm a model is loaded before sending requests. Use the upstream TabbyAPI setup guide for installation and configuration.

Compatibility note: MiniCPM-V 4.7 and dSparK support is in the vcruz305/exllamav3 fork; stock ExLlamaV3 support depends on the installed release supporting the MiniCPM-V 4.7 architecture and multimodal path. Confirm this on the intended version before deployment; the interactive example above does not itself establish image/video support.

Benchmarks and measurements

Official upstream benchmarks

The values in this section are OpenBMB's published results for the original checkpoint, transcribed from the charts in the official MiniCPM-V 4.7 card. They were not rerun on this EXL3 release and must not be interpreted as quantized-model scores.

OpenBMB also reports 13+ tokens/s on an iPhone 17 Pro Max for its on-device stack. That number uses OpenBMB's own runtime, scheduling, quantization and speculative-sampling setup; it is not an ExLlamaV3 performance result.

Vision-language — upstream 35B-A3B Instruct

MiniCPM-V 4.7 supports selectable 4× and 16× visual-token compression. Higher is better unless marked with ↓.

Language and tool use — upstream 35B-A3B Instruct

Measurements for this quantization

The quality rows below were run twice on the exact 19-shard shipping pack against immutable BF16 final-logit stores. The pooled row contains 20,480 equally weighted tokens.

The throughput workload used a production TabbyAPI/ExLlamaV3 server, 1,024 uncached prompt tokens, 256 generated tokens, greedy decoding, concurrency 1, two warmups and seven measured requests. Every measured prompt and completion had the intended token count. Peak CUDA allocation was 20.33 GiB in the 8,192-token text-quality configuration; the separate 4,096-token text-plus-vision smoke peaked at 18.33 GiB.

Re-scored on the fork's current master runtime (the one that enables one-pass dSparK verification): pooled top-1 agreement 0.934277 and mean KL 0.031462 against the same BF16 references. The Sixcat and throughput rows above were measured on the earlier release runtime.

Sixcat used v0.7.0 with 20 first-scored responses per category, custom temperature 0.7, seed 1, thinking enabled, host-guarded code execution and concurrency 4. All 120 items were scored with no truncations or low-confidence parses; two instruction rows were flagged as loop failures.

Full 262,144-token context

One cold 261,886-token request with four passcodes at 10/35/60/90% depth, then decode until the context is exactly full (261,888 + 256 = 262,144). TabbyAPI, Q4 KV cache, target-only, one request at a time.

On the Spark, Q4 KV is both smaller and faster at full context than FP16 KV (27.26 tok/s). On the 4090, decode stays at 84–91 tok/s from 2K to 262K tokens.

dSparK speculative decoding

The pinned official OpenBMB dSparK checkpoint is included under dspark/. It runs with the vcruz305/exllamav3 fork (master), which verifies a whole draft window in one target pass and produces byte-identical output to target-only greedy decoding.

Same workload as above: 1,024-token prompts, 256 generated tokens, greedy, concurrency 1, two warmups and seven measured requests. The gain depends on the content: code and math prompts accept more drafted tokens than open-ended chat.

dSparK only helps short prompts. Draft acceptance falls as the prompt grows, and below the break-even point it is slower than target-only decoding (output stays identical):

On an RTX 4090 the only measured gain is the 1,024-token workload (92.06 → 103.14 tok/s, +12%). Use dSparK for short chat on a Spark; use target-only for long documents and on the 4090. The recipes enforce this (dSparK is opt-in and capped at 8,192 context).

To enable it in TabbyAPI:

draft_model:
  draft_mode: model
  draft_model_dir: /path/to/MiniCPM-V-4.7-35B-A3B-EXL3-SAGE-3.74bpw
  draft_model_name: dspark
  draft_num_tokens: 7

Draft windows are capped at 7 tokens, the largest window the fork verifies exactly. Set EXL3_DSPARK_SERIAL_VERIFY=1 to use the slower token-by-token reference verifier instead. With a Q4/Q6/Q8 KV cache use fork master at or after 475461e9; earlier builds crash on the first speculative step.

Source and provenance

The base model is the public OpenBMB MiniCPM-V 4.7 35B-A3B checkpoint. The bundled dspark/ directory comes from openbmb/MiniCPM-V-4.7-35B-A3B-DSpark, pinned at revision 5facb75c251ffd524524a544c1f5fd97ba0d6c5c. Its model.safetensors SHA-256 is 211eda7b962795ba727b0d7db072d8c474dc3b94075f85f123fe363caf5e9679.

Acknowledgements

DSpark config validation, conversion passthrough of the Markov and confidence heads, and the draft-pack tooling in the fork are by r0b0tlab, merged with permission.

Limitations

This is a quantized model. Quantization may change output quality, including vision quality. Actual memory use, throughput, supported context length and multimodal behavior depend on the compatible runtime, hardware and final pack. Test the exact files and runtime configuration for your use case.

This model is provided without warranty. Review the upstream license and model documentation, and apply appropriate safeguards to generated outputs.