A DSpark draft (speculative decoding) head for diffu-minicpm-ocr-htr, the Swedish handwriting reader of diffu. It proposes up to 15 tokens at a time for the reader to verify, so the reader decodes faster and, decoding greedily, writes the same text. It is trained the way OpenBMB trained MiniCPM-OCR-1.0-DSpark, but on this reader's own answers.
The files are in the speculators format (config.json, its config.py
and the weights), and the config names Gabriel/diffu-minicpm-ocr-htr as the verifier.
The draft runs inside vLLM, beside the reader. It fits this reader only: a draft learns its target's own answers.
You need Linux, an NVIDIA GPU with about 20 GB free, and uv. MiniCPM-OCR needs a vLLM nightly from 29 October 2026 or later (vllm-project/vllm#64105) with transformers 5.22.1; this is the build the draft was trained and tested with:
uv venv --python 3.12 htr && source htr/bin/activate
echo "transformers==5.22.1" > overrides.txt
uv pip install --override overrides.txt openai \
"vllm @ https://wheels.vllm.ai/fa3b9d1377219247060d3c8b49eae199c9e7e154/vllm-0.33.0rc1.dev234%2Bgfa3b9d137-cp38-abi3-manylinux_2_28_x86_64.whl"
On an ARM machine (such as a DGX Spark), use the ..._aarch64.whl of the same commit. While the repos are private,
log in first with hf auth login.
Serve the reader with the draft:
vllm serve Gabriel/diffu-minicpm-ocr-htr --served-model-name htr --max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "video": 0}' \
--mm-processor-kwargs '{"downsample_mode": "4x", "max_slice_nums": 9}' --trust-request-mm-kwargs \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--speculative-config '{"method": "dspark", "model": "Gabriel/diffu-minicpm-ocr-htr-dspark", "num_speculative_tokens": 15, "enable_adaptive_verification": false}'
Then ask it exactly as you ask the reader alone: one line or text-region image and the prompt Text Recognition:,
decoded greedily.
import base64
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
image = base64.b64encode(open("line.jpg", "rb").read()).decode()
reply = client.chat.completions.create(
model="htr",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image}"}},
{"type": "text", "text": "Text Recognition:"}]}],
max_tokens=512, # 4096 for a text region
temperature=0.01, top_p=0.001, extra_body={"top_k": 1, "repetition_penalty": 1.0},
)
print(reply.choices[0].message.content)
Leave out --speculative-config to serve the reader alone. To check that the draft is used, read vLLM's
spec_decode_num_* counters at http://localhost:8000/metrics. They rise with every request, and the acceptance
length is 1 + accepted tokens / drafts.
The reader is a hybrid model (linear-attention layers), so with a draft vLLM keeps one copy of each request's recurrent state per draft token (speculators: throughput vs. interactivity): each request takes more memory, and fewer fit in a batch. Measure with and without the draft at the concurrency you read at.
On held-out answers: 8.65 tokens accepted a step on average, acceptance rate 85.1%.
Measured on 9 November 2026 on the same machine, one RTX PRO 6000 Blackwell per model, with the vLLM build under Setup, prefix caching off and greedy decoding. The test data is the first 1,000 of 5,000 test lines (whole volumes held out of training) and 200 dev regions. The reference is the Swedish Lion TrOCR (float32, greedy) on the same lines and GPU; it reads single lines only. Figures are lines or regions read per second.
| At a time | TrOCR | Reader alone | Reader + draft | Draft speedup | |
|---|---|---|---|---|---|
| Lines | 1 | 4.3 | 13.3 | 18.2 | 1.37× |
| Lines | 64 (TrOCR: batch 32) | 6.5 | 74.6 | 82.2 | 1.10× |
| Regions | 1 | – | 1.95 | 4.08 | 2.09× |
| Regions | 64 | – | 12.4 | 13.2 | 1.07× |
The acceptance length in vLLM is 7.5 on lines and 9.7 on regions, out of at most 16 (from vLLM's spec_decode counters).
The draft helps most when few requests run at once: reading a page while someone waits, or long regions. With 64
requests at a time the GPU is already busy, and the gain is about 10%.
The draft does not change what the reader writes. It only proposes tokens; the reader accepts or rejects each one.
| CER on the same text (OpenBMB's corpus CER) | TrOCR | Reader alone | Reader + draft |
|---|---|---|---|
| 1,000 test lines | 4.48% | 4.99% | 4.97% |
| 200 dev regions | – | 4.28% | 4.27% |
964 of the 1,000 line readings with the draft are character-for-character identical to the reader's alone. The rest differ by GPU rounding, not by the draft: the reader alone gives the same rate (959 of 1,000 identical) when it reads the same lines one at a time and then 64 at a time. On these 1,000 lines TrOCR is the more accurate of the two readers. The reader's own card has the full comparison.
runs/dspark-all/ holds the TensorBoard log of the training run: the loss, the acceptance length and each draft
position's accuracy every 10 steps, and the held-out validation after each epoch.
Apache-2.0, as its base model.
Gabriel/diffu-minicpm-ocr-htr