openbmb/MiniCPM-OCR-1.0-GPTQ

Demo (CN / Global) · GitHub · Free API · SDK usage · DSpark · Skill

This repository hosts the GPTQ W4A16 quantized version of MiniCPM-OCR 1.0. For the original BF16 weights and the base model card, see openbmb/MiniCPM-OCR-1.0. Everything below describes the model itself; see Quantization for what changed. MiniCPM-OCR 1.0 (1.3B) is the first OCR-specific model in the MiniCPM-V/o family. It focuses on text-centric document understanding — text, tables, formulas, charts, seals, and structured information extraction — and balances quality against speed on real-world captures.

Highlights

• 🏆 SOTA document parsing: 93.68 on Real5-OmniDocBench, a benchmark for real-world document parsing, with leading results on OmniDocBench v1.6, Wild-OmniDocBench, and olmOCR-Bench. Handles Chinese and English documents and PDFs, including scans, curved pages, photographs, skewed pages, and recaptured documents.
• ⚡ Efficient inference. With DSpark decoding acceleration (BF16, GGUF) end-to-end processing efficiency improves by 55%. Throughput is about 1.49× PaddleOCR-VL-1.6 and 3.62× MinerU2.5 Pro.
• 🔍 General recognition and information extraction: 54.7 on OCRBench v2, plus key information extraction (KIE) for downstream automation.
• 🛠️ Easy to deploy and develop: vLLM, SGLang, llama.cpp, and Ollama from day one, with quantized variants and multi-chip builds. Supports further fine-tuning, and ships a pip SDK, a full API, and agent workflows. Online demos: China · Global.

SOTA quality at higher throughput

Among the open-source OCR models compared, MiniCPM-OCR 1.0 reaches 1.2 pages/s and the highest equal-weight average score across OmniDocBench v1.6, Real5-OmniDocBench, Wild-OmniDocBench, and olmOCR-Bench.

_{Average score is the equal-weight mean of the four benchmarks. All models use accelerated configurations. Settings and reproduction: reproduction guide. Per-model throughput: model guide.}

Benchmark comparisons

Full MiniCPM-OCR 1.0 evaluation results

Each panel keeps its own benchmark's scores; higher is better. Scores from different benchmarks are not directly comparable. Test settings and reproduction steps are in the reproduction guide; per-benchmark discussion is in the model guide.

Throughput

On an NVIDIA GeForce RTX 5090 with all 1651 OmniDocBench v1.6 images, image-level concurrency 1, and each system's accelerated configuration, MiniCPM-OCR 1.0 reaches 1.2 pages/s (relative throughput 1.00×; higher is faster).

Concurrency 1 is image-level: regions inside a page are still recognised in parallel. Real throughput also depends on page content, backend, concurrency, and hardware.

Examples

Left to right: original document, detected layout, and rendered Markdown.

English documents

Chinese documents

More examples

Original documents and parsed results, side by side.

English documents

Chinese documents

Model architecture

Parsing pipeline

Full-page parsing runs as a two-stage pipeline orchestrated by the minicpm-ocr SDK. The two stages use separate models, run in separate environments, and may run on different machines:

• Stage 1 — Layout detection (SDK environment). The PP-DocLayoutV3 layout model detects document regions — text, tables, formulas, charts, seals — and determines reading order; the SDK crops each region. It runs locally inside the SDK environment and requires PaddlePaddle / PaddleX.
• Stage 2 — Region recognition (inference service). The SDK sends each cropped region over HTTP to a MiniCPM-OCR 1.0 inference service — served by vLLM, SGLang, llama.cpp, or Ollama — with a prompt matching the region type, then assembles the recognized content into the final Markdown / JSON in reading order.

See the model guide for more on architecture, parsing workflow, deployment options, and benchmark breakdowns.

Quick start

Use with the MiniCPM-OCR SDK

Required before using either the CLI or Python API: deploy and start the OCR model service first, and wait until it is ready to accept requests. The SDK examples below are clients of that running service; they do not deploy or start the OCR model service automatically.

Use the official minicpm-ocr SDK to parse local images or PDFs with this GPTQ checkpoint. The SDK detects layout with PP-DocLayoutV3, recognises each region, and saves structured JSON and Markdown in reading order.

Install Python 3.10+ and the SDK:

pip install "minicpm-ocr[all]"

Install minicpm-ocr for your platform using the installation guide.

Prerequisite: deploy and start the model service

Before running the SDK, start a vLLM recognition service for the GPTQ checkpoint in this repository, openbmb/MiniCPM-OCR-1.0-GPTQ, following the vLLM deployment guide. Use this repository ID as the served model path, --served-model-name minicpm-ocr, --port 8080, and --max-model-len 40960. The service reads the GPTQ quantization settings from this repository's config.json.

The SDK examples use http://127.0.0.1:8080, model alias minicpm-ocr, the vllm backend, and an NVIDIA GPU for layout detection. Match the URL, alias, and context length to your service. The SDK connects to the running service; it does not load the quantized weights into the SDK process.

CLI

Start the model service described above and wait until it is ready before running this command. Save a local image as demo.png, then run:

minicpmvocr parse demo.png \
  --backend vllm \
  --server-url http://127.0.0.1:8080 \
  --model minicpm-ocr \
  --device cuda:0 \
  --set engine.max_model_len=40960 \
  --set run.run_root=output \
  --output output/demo.md

The CLI automatically downloads PP-DocLayoutV3 to models/PP-DocLayoutV3_infer/ when needed. The final Markdown is saved to output/demo.md; structured JSON and per-page Markdown are kept in output/<run_tag>/results_json/ and output/<run_tag>/output/.

Python API

The model service described above must already be running before you execute this Python example. Set engine.server_url and engine.model_name to match that service.

The same pipeline is available from Python:

from importlib.resources import files
from pathlib import Path

from minicpmvocr import load_config, pipelined_run
from minicpmvocr.layouts.download import ensure_layout_model


def main():
    layout_model = ensure_layout_model(Path("models/PP-DocLayoutV3_infer"))

    cfg = load_config(
        str(files("minicpmvocr").joinpath("configs/minicpmvocr.yaml")),
        overrides={
            "input": {"file_path": str(Path("demo.png").resolve())},
            "engine": {
                "type": "vllm",
                "server_url": "http://127.0.0.1:8080",
                "model_name": "minicpm-ocr",
                "max_model_len": 40960,
            },
            "layout": {
                "model_dir": str(layout_model),
                "device": "cuda:0",
            },
            "pipeline": {"overlap_layout": True},
            "run": {"run_root": str(Path("output").resolve())},
        },
    )

    pipelined_run(cfg)

    print("JSON directory:", cfg["_meta"]["result_json_dir"])
    print("Markdown directory:", cfg["_meta"]["output_dir"])


if __name__ == "__main__":
    main()

ensure_layout_model() reuses or downloads the official layout model. Results are written to output/<run_tag>/results_json/ and output/<run_tag>/output/. To parse a PDF, replace demo.png with its local path.

For CPU layout detection, use --device cpu in the CLI or set layout.device to cpu in Python. This changes only the layout detector; the recognition service still uses the deployed quantized model.

Run minicpmvocr parse --help for CLI options, or see the SDK API reference for configuration and document-level Markdown export.

Prompts

The prompt selects the task and must match the region you cropped:

If finish_reason is length, the output was truncated — raise the token budget.

Key information extraction (KIE)

MiniCPM-OCR 1.0 can pull specified fields straight out of an image — for example the store name and amount on a receipt. Send the image with a field-extraction prompt and ask for a JSON object; you do not need to produce page Markdown first.

Use the image recognition API example and replace Text Recognition: with:

Extract the "store_name", "amount". Format the output as a JSON object with keys: "store_name", "amount". If an item is not in the image, use "###".

Replace both field lists with your own field names (Chinese or English). The response is model-generated text: parse the JSON and check fields and values before using it downstream.

Agent skill

Have a coding agent follow the skill to set up the environment, download the model, and parse a first document:

Read and execute https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md

Fine-tuning

LLaMA-Factory · ms-swift — data preparation, full-parameter SFT with the multimodal projector frozen, training, and evaluation (French handwritten text lines).

Quantization

Only the language-model linear layers are quantized. The vision encoder, multimodal projector, lm_head, and the GatedDeltaNet in_proj_a / in_proj_b projections stay in BF16, so image encoding is unchanged and recurrent gating is preserved.

Weight footprint drops from about 2.42 GiB (BF16) to about 1.74 GiB.

quantization_config in config.json is what vLLM reads; no extra flag is needed. quantize_config.json records the quantization parameters.

Related checkpoints

DSpark requires a matching main and draft model from the same release — do not mix draft weights with a different main checkpoint. See DSpark acceleration.

For OCR quality assessment (auto-accept, human-review routing, best-of-N across models), see MiniCPM-OCR Jev.

FAQ

Installation, model downloads, GPU memory, input formats, and unexpected outputs: FAQ.

License

Model weights, code, and documentation are released under the Apache License 2.0.