Unlimited-OCR-2 — GGUF with a Russian-document imatrix

GGUF builds of baidu/Unlimited-OCR-2 (3.3B MoE, MIT). Upstream quants are calibrated on Chinese/English document benchmarks; the Q4_K_M here is calibrated on 92 pages of real Russian documents.

Files

Verified

Same page, greedy, BF16 vs Q4_K_M, on documents not in the calibration set:

Character-level equality after normalising layout boxes ([x1,y1,x2,y2] -> [BOX]); box coordinates matched exactly. Timing is a T4.

BF16 GGUF vs the vendor reference (UnlimitedOCRForCausalLM.infer(), original remote code, same page, greedy): 99.94% char similarity, identical block sequence, box coordinates differ by ±1 (bf16/f32 rounding).

Why a patch is required

unlimited-ocr2 is not supported by mainline llama.cpp (that one handles Unlimited-OCR v1 / DeepSeek-OCR). Two differences a plain conversion cannot capture:

1. page-2D RoPE — the position is (offset inside the page, page parity), not the sequence index: 63 frequency pairs drive the token axis, 1 pair drives the page axis.
2. DASH / SR-SWA attention — every token attends a token-level local window (128) plus up to top_k = 12 remote blocks of 32 tokens, selected per layer by a router (head_cls block summaries -> entmax-1.5 prior -> additive log bias).

Converted as v1 (full attention + 1D RoPE) the model loads and runs but silently produces the wrong document structure.

Build

Pinned to llama.cpp b11913 (commit 7aca0d2ea8):

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp && git checkout b11913
git apply /path/to/llama.cpp-patch/unlimited-ocr2.patch
git apply /path/to/llama.cpp-patch/no-repeat-ngram.patch   # optional but recommended

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_EXAMPLES=OFF \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-imatrix llama-quantize

On Colab, skip the compile with binaries/uocr2-llama-cpp-cuda75.tar.gz (BINARIES.md has the sha256, flags and the caveat that it is sm_75 only). That tarball was built before the sampler patch, so it has no --no-repeat-ngram-size; rebuild if you want the guard.

Run

llama-server -m uocr2-Q4_K_M.gguf --mmproj mmproj-uocr2-f16.gguf \
  -ngl 99 -c 4096 -np 1 --temp 0 --port 8099 \
  --no-repeat-ngram-size 24 --no-repeat-ngram-window 1024

Prompt format, exactly as trained — <PAGE> before the image:

<PAGE><image>Multi page parsing.

Multi-page: repeat the pair per page in page order. Image first produces garbage.

Decoding caveat: the no-repeat-ngram guard

Without a guard the model can fall into a token loop on dense historical tables and never emit </BLOCK>: greedy decoding then runs to max_tokens and the truncated answer still looks complete. Reference serving (SGLang) prevents this with a no_repeat_ngram sampler; mainline llama.cpp has no such sampler, only the repetition/DRY penalties, and those damage legitimately repeated table content.

llama.cpp-patch/no-repeat-ngram.patch adds it, as a first-class sampler with a public API, a CLI flag and an HTTP field:

llama-server ... --no-repeat-ngram-size 24 --no-repeat-ngram-window 1024
# or per request: {"no_repeat_ngram_size": 24, "no_repeat_ngram_window": 1024}

Semantics are identical to transformers/SGLang: keep the generated-token history, and if the last n-1 tokens already occurred in it, ban every token that followed that occurrence. Exact key comparison, so it can never ban a token by accident; the whole candidate set is never banned; DRY and the penalties are untouched.

Measured on this model (Q4_K_M, Metal, greedy, 12 real pages):

n=24 is the recommended value for this checkpoint: it never fires on clean pages (so the output is exactly the unguarded output) and it is tight enough to break the table loop after a couple of repeats. The window has to be larger than the repeat period: n=24, window=256 still fixed that page (1509 tok), n=35, window=128 did not (3187 tok, runaway), so 1024 is the safe default; multi-page prompts need 4096. Do not substitute DRY: it makes output worse.

Quantization recipe

llama-quantize --imatrix uocr2-ru.imatrix.gguf \
  --tensor-type 'blk\.\d+\.indexer\..*=F32' \
  --output-tensor-type Q8_0 --token-embedding-type Q8_0 \
  uocr2-bf16.gguf uocr2-Q4_K_M.gguf Q4_K_M

• blk.*.indexer.* — the DASH router; must stay F32, quantizing it degrades block selection.
• attention, shared experts, token embeddings and output -> Q8_0.
• routed experts -> Q4_K/Q5_K mixture (Q4_K_M).
• Q8_0 build: plain llama-quantize ... Q8_0 (8.52 BPW, 3.0 GB).

Calibration corpus

92 real Russian pages, published with provenance: p4ulbr4dl3y/uocr2-ru-imatrix-calib

Pipeline: BF16 model greedy OCR per page -> target texts -> multimodal llama-imatrix (the patch teaches it --mmproj) -> llama-quantize. ~6 min on a T4.

Credits

Architecture and weights: Baidu. The llama.cpp port is RX5950XT/Unlimited-OCR-2-GGUF (unlimited-ocr2.patch), redistributed under the llama.cpp MIT license unchanged. Calibration corpus and ru-tuned imatrix: this repo.