English | 简体中文

OvisOCR2 GGUF

GGUF builds of ATH-MaaS/OvisOCR2 for llama.cpp. OvisOCR2 is a small (Qwen3.5-0.8B based) document parser that turns a page image into Markdown with LaTeX formulas and HTML tables.

Converted and quantized with llama.cpp v0.5.0 (7fe450e). Eight language model / projector pairs were evaluated end to end on all 1,651 OmniDocBench pages. This is an unofficial conversion.

Which files to download

You need one language model and one mmproj (vision projector). Use mmproj-OvisOCR2-Q8_0.gguf unless you have a reason not to.

Avoid Q2_K: it loses about 9 points (87.30) and runs into the output-token limit on 12% of pages.

† IQ models use an importance matrix calibrated on OmniDocBench outputs. Their scores overlap with the calibration data and are not clean held-out results.

Quick start

Download a pair and the recommended prompt:

hf download MDfox/OvisOCR2-GGUF-SOTA-20260823 \
  OvisOCR2-Q6_K.gguf mmproj-OvisOCR2-Q8_0.gguf prompt.txt --local-dir .

Single image:

llama-mtmd-cli -m OvisOCR2-Q6_K.gguf --mmproj mmproj-OvisOCR2-Q8_0.gguf \
  --image page.png -f prompt.txt \
  -ngl all -c 32768 -n 16384 --temp 0 --flash-attn on \
  --image-min-tokens 196 --image-max-tokens 8100

OpenAI-compatible server (the setup used for the benchmark):

llama-server -m OvisOCR2-Q6_K.gguf --mmproj mmproj-OvisOCR2-Q8_0.gguf \
  --jinja -ngl all -c 98304 --parallel 3 --kv-unified --flash-attn on \
  --cache-type-k bf16 --cache-type-v bf16 \
  --image-min-tokens 196 --image-max-tokens 8100

Send the page image followed by the text of prompt.txt as one user message, with temperature: 0, max_tokens: 16384 and "chat_template_kwargs": {"enable_thinking": false}.

The files were produced with llama.cpp v0.5.0. Earlier tests also loaded the non-IQ files on v0.3.0 and v0.4.0 servers; newer releases are expected to work but are not covered by these benchmarks.

Files

Every quantized language model comes straight from the BF16 conversion, and the Q8_0 projector straight from the BF16 projector; nothing was re-quantized twice. Hashes are in checksums.sha256.

OmniDocBench results

All 1,651 pages, official end-to-end quick-match evaluator. No page failed in any run.

Overall = ((1 − text edit) × 100 + formula CDM + table TEDS) / 3. Sizes are the two files combined, not VRAM. "Hit token limit" counts pages that reached 16,384 output tokens, usually because of repetition.

What the numbers say:

• Q8_0, Q6_K and BF16 are within 0.06 points of each other, so the 8-bit and 6-bit files are effectively lossless.
• Going down to Q4_K_M or Q3_K_S costs 0.5–0.8 points, mostly on tables and formulas.
• Below 3 bits quality drops fast. Q2_K repeats itself far more often, which also makes it the slowest file.

Real-world capture (Real5-OmniDocBench)

Real5-OmniDocBench re-captures OmniDocBench pages under five real-world conditions (1,355 pages each). Only IQ3_XXS + Q8_0 mmproj was run here.

Scanned and unevenly lit pages are close to the clean-page score; skewed pages hurt tables the most. The † calibration caveat applies here too.

Benchmark setup

• Data: OmniDocBench at revision aa1ee96, 1,651 pages; Real5-OmniDocBench at revision b3ca1d3.
• Scoring: official end-to-end quick-match evaluator, run in the fixed Docker image ghcr.io/zeng-weijun/omnidocbench-eval:repro-ubuntu2204.
• Inference: llama.cpp v0.5.0 llama-server, CUDA, one RTX 5060 Ti 16 GB. 3 parallel slots, 32K context per slot, BF16 KV cache, flash attention, temperature 0, up to 16,384 output tokens, 196–8,100 image tokens, prompt from prompt.txt.
• Evidence: benchmark-results/ has per-run summaries, evaluator outputs, settings, GPU traces and the scripts used (reproduction/). summary.csv lists every metric of every run. Raw files keep the local paths recorded at run time.

About the IQ files

The IQ2/IQ3 files share one importance matrix computed from cleaned OmniDocBench predictions of the BF16 model. That makes their OmniDocBench and Real5 scores optimistic. If you need a fair number for these files, evaluate them on your own documents. Details: CONVERSION_NOTES.md and calibration/.

Changes from the previous release

The previous release (2026-09-12) was built with llama.cpp v0.4.0. This update:

• rebuilds every file with v0.5.0, which adds one metadata key; tensor data of the BF16 model is unchanged;
• adds IQ3_S, IQ3_XS and four IQ2 variants, and recalibrates the IQ files from BF16 instead of Q8_0 predictions;
• re-runs the OmniDocBench evaluation on v0.5.0 and adds Q6_K, BF16, IQ2_M and Real5 results. Q3_K_M, IQ3_M and Q8_0 + BF16 mmproj were not re-run; on v0.4.0 they scored 96.20, 95.95 and 96.45.

The v0.4.0 files and results remain available in the repository's commit history.

License

Apache-2.0, following the original model. Please cite the OvisOCR2 authors when you use this model.