MiniCPM-V-4.7-35B-A3B EXL3 (4.15 bpw mixed, RTX 3090 target)

EXL3 quantization of openbmb/MiniCPM-V-4.7-35B-A3B with a per-tensor mixed bitrate recipe derived from measured end-to-end KL attribution, sized to fit and run at full context on a single 24 GB RTX 3090.

• Quantizer: exllamav3 1.6.1 (mul1 codebook), LDLQ compensation on a mixed web/code/multilingual calibration corpus (250 x 2048 tokens).
• Text backbone (Qwen3.5 hybrid GDN/full-attention MoE): mixed K per tensor from KL attribution, hq MoE promotion, 4.15 bpw average.
• Vision tower (SigLIP NaViT + window merger + downsample merger): unquantized BF16 — the tower is not validated for low-bpw EXL3, and at ~0.5 B params the BF16 copy costs ~1 GB.
• lm_head: 6 bpw; token embedding: 16 bpw.
• Every checkpoint tensor and processor asset is preserved: 4x/16x downsampling, slice-based multi-crop, video, canvas M-RoPE. Apache-2.0.

Verified

• Text generation: greedy smoke tests (factual + code prompts) produce coherent output; full 8-workload spec-decode benchmark (6264 verification rounds, greedy) runs clean.
• Per-tensor bitrates live in quantization_config.json.
• Vision tower forward parity and multimodal (image/crop/video) E2E: not yet validated for this pack — the tower is BF16 so parity is expected but unmeasured. Text-only use is verified.

Speculative decoding (DSpark drafter, same org)

Pair with MiniCPM-V-4.7-35B-A3B-DSpark-EXL3-8.0bpw for block-parallel speculative decoding. Measured on the RTX 3090, greedy, 8-workload mix (agentic/coding/creative/translation/repetition), block 15:

Per-category the scheduler is bimodal: Coding 223 t/s (1.53x), trivial repetition 398 t/s (2.72x); translation rounds mostly skip drafting (0.64x — disable spec-decode for pure translation workloads). Scheduler: confidence-head STS calibration + Theta verification-length selection, global-argmax variant (see exllamav3).

In this repository

• CONVERT.md - full conversion record (source revision, exact quantiser command, method, timings)
• calibration.md - calibration data and caveats
• quant_layers.json - per-layer quantisation error metrics
• SERVE.md - the Ampere serving recipe (engine pin, memory budget, speculative decoding, scheduler)
• UPLOAD_MANIFEST.json - sha256 manifest of every file

Usage

from exllamav3 import Config, Model, Cache, Tokenizer, Generator
config = Config.from_directory("MiniCPM-V-4.7-35B-A3B-EXL3-4.15bpw-hq")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens = 32768)
model.load()
tokenizer = Tokenizer.from_config(config)
generator = Generator(model = model, cache = cache, tokenizer = tokenizer)
print(generator.generate(prompt = "Once upon a time", max_new_tokens = 200))

Requires a recent PyTorch (CUDA 12.4+) and exllamav3 >= 1.6.1.