MiniCPM-V-4.7-35B-A3B-GGUF-Serve-mixed-4-8bit
A llama.cpp GGUF of MiniCPM-V 4.7 (35B total / 3B active MoE), sized so that the weights plus a 128K f16 KV cache fit on a 32 GB GPU with plenty to spare. Built and measured on an AMD Radeon AI PRO R9700 (32 GB, Vulkan).
The language model is a hybrid: only 10 of its 40 layers use full attention, so a 131K KV cache is about 2.5 GiB.
File size: 20.61 GiB, 5.11 BPW. Recommended VRAM: 24 GB or more for the full 131K context.
This file is the language model only. For image input, add an mmproj from bartowski/MiniCPM-V-4.7-35B-A3B-GGUF.
Recipe
Quantized with llama-quantize (llama.cpp b11836) from bartowski's BF16 conversion, using his imatrix.
| Tensors | Type |
|---|---|
Routed experts gate/up (ffn_{gate,up}_exps) |
Q4_K |
Routed experts down (ffn_down_exps) |
Q5_K |
| Everything else quantizable: attention, linear-attention, shared expert, embeddings, LM head | Q8_0 |
| Routers, norms | F32 (unquantized) |
llama-quantize --imatrix imatrix.gguf --tensor-type-file tensor-types.txt \
MiniCPM-V-4.7-35B-A3B-bf16-00001-of-00002.gguf MiniCPM-V-4.7-35B-A3B-GGUF-Serve-mixed-4-8bit.gguf Q8_0
tensor-types.txt:
ffn_(gate|up)_exps=q4_k
ffn_down_exps=q5_k
llama-server
llama-server -m MiniCPM-V-4.7-35B-A3B-GGUF-Serve-mixed-4-8bit.gguf -ngl 99 -fa on -c 131072 -ub 512 --jinja
Results (R9700, Vulkan, llama.cpp b11836)
With the full 131,072-token f16 KV cache, the GPU uses 23.9 GiB dedicated at -ub 512 and 24.4 GiB at -ub 2048 (Windows GPU counters, desktop included). Nothing spills to system memory.
llama-bench -fa 1 -p 2048 -n 128 -d 0,32768,131072, tok/s:
| ubatch | depth | prefill (pp2048) | decode (tg128) |
|---|---|---|---|
| 512 | 0 | 4,473 | 144.3 |
| 512 | 32K | 2,889 | 120.9 |
| 512 | 131K | 1,360 | 86.2 |
| 2048 | 0 | 4,826 | 144.1 |
| 2048 | 32K | 3,046 | 121.1 |
| 2048 | 131K | 1,420 | 86.5 |
Long-context retrieval: a 99,399-token prompt (repository source and docs, with a passphrase planted in the middle) through llama-server's router. Prefill ran at 2,319 tok/s, decode at 91.7 tok/s, and both answers were correct. Sent again, it reused 99,395 cached prompt tokens.
Known quirk: on that long prompt the model answered inside its thinking block and never closed it, so llama-server returned the answer in reasoning_content with an empty content. Short prompts separate thinking and answer normally. Pass "chat_template_kwargs": {"enable_thinking": false} when you want direct answers.