Xiaomi-OCR-0 · NInfer

简体中文

Model overview

A BF16 NInfer version of Xiaomi-OCR-0, based on Qwen3.5-0.8B-Base, for OCR, document parsing, table extraction and formula recognition. It runs with NInfer Extended, using native C++/CUDA kernels for vision encoding, prefill and decode.

The model file includes language, vision and MTP weights, the tokenizer and image-processing configuration. MTP1 is recommended; the same file also supports ordinary decoding and MTP3.

Performance

Prefill and decode

Warm-run medians, BF16 weights and KV, one request, 4K context and greedy decoding. Transformers uses compiled vision, prefill and decode, with CUDA Graph for decode. Prefill includes vision encoding and language processing. The formula sample generates 256 tokens.

Complete request latency

Includes image processing, inference and the local HTTP request.

NInfer BF16 without MTP vs. MTP1

Paired warm-run medians from the same NInfer build, with BF16 weights and KV and a 4K context.

Single-request prefill and decode

Complete single-request latency

Four-request throughput

Total output tokens across four concurrent requests divided by complete request time, including image processing, prefill, decode and HTTP overhead.

Transformers MTP1 comparison

The Transformers MTP1 reference uses a compiled PyTorch MTP adapter with the same proposal head as NInfer. See the MTP comparison guide for its implementation.

Accuracy

The following results hold on both RTX 5070 Ti and RTX 6000D. CER is measured against human annotations; MTP output agreement and character differences are measured against ordinary NInfer decoding.

The Chinese document, English contract and 256-token formula outputs used in the speed tests match Transformers exactly on both GPUs. They also match the Transformers MTP1 reference.

MTP acceptance

Accepted draft tokens on complete pages; results are identical on both GPUs.

MTP1 improves all measured single-request and four-request workloads and is the recommended default.

Usage

Build

Requirements: Linux or WSL2, CUDA with sm_120a support, cuDNN 9 and cuBLAS. See the build guide for dependencies.

git clone https://github.com/ByronLeeeee/ninfer-extended.git
cd ninfer-extended
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCUDNN_ROOT=/path/to/cudnn -DCUBLAS_ROOT=/path/to/cublas
cmake --build build --target ninfer-serve -j

Download and start

pip install huggingface_hub
hf download ByronLeeee/Xiaomi-OCR-0-Ninfer \
  xiaomi-ocr-0-bf16-MTP.ninfer --local-dir models
./build/apps/ninfer-serve models/xiaomi-ocr-0-bf16-MTP.ninfer \
  --host 127.0.0.1 --port 8080 --model-id ocr --vision \
  --max-context 32768 --kv-capacity 131072 --max-concurrency 4 \
  --kv-dtype bf16 --prefill-chunk 1024 --no-thinking --greedy \
  --spec mtp --draft-tokens 1 --lm-head-draft

Use --prefill-chunk 2048 on RTX 6000D. For MTP3, set --draft-tokens 3. For ordinary decoding, remove --spec mtp --draft-tokens 1 --lm-head-draft.

Recognize an image

Save the input as page.png and run:

import base64
import json
import urllib.request
from pathlib import Path

image = base64.b64encode(Path("page.png").read_bytes()).decode()
body = {
    "model": "ocr",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64," + image}},
        {"type": "text", "text": "Extract the text in the image."}
    ]}],
    "temperature": 0,
    "max_tokens": 4096,
}
req = urllib.request.Request(
    "http://127.0.0.1:8080/v1/chat/completions",
    data=json.dumps(body).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as response:
    print(json.load(response)["choices"][0]["message"]["content"])

Change the prompt for text, tables, formulas or full-page parsing. More examples are in the OCR guide.

Model files

License: Apache-2.0.

Full results · Transformers MTP results · Artifact manifest