Unlimited-OCR — GGUF

GGUF quantizations of baidu/Unlimited-OCR, a 3B vision-language OCR model that pushes DeepSeek-OCR one step further (one-shot, long-horizon document parsing). This repo contains a full spread of K-quants and i-quants of the language model plus the vision projector (mmproj) needed for image input.

⚠️ Requires a DeepSeek-OCR–aware llama.cpp build (PR #17400). Unlimited-OCR uses the DeepSeek-OCR architecture (a SAM+CLIP DeepEncoder vision tower with a DeepSeek-V2 MoE text decoder). Support is not yet merged into upstream main — stock llama.cpp will not load these files. Build the PR branch (instructions below).

Files

Every run needs two files: one language model GGUF (pick a quant) plus the shared vision projector. The projector is fp16 and identical for all quants.

Vision projector (required for all of the above):

Sizes are the on-disk GGUF sizes. The vision encoder is kept at F16 (not quantized) — it is small and quantizing it hurts OCR accuracy. i-quants were built with an importance matrix (imatrix) computed from a general-text calibration set.

Build llama.cpp with DeepSeek-OCR support

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/24975/head:pr24975 && git checkout pr24975
cmake -B build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON for NVIDIA
cmake --build build -j --target llama-mtmd-cli llama-server

Quick start

Download one quant + the projector (you always need both):

huggingface-cli download sahilchachra/Unlimited-OCR-GGUF \
  --include "Unlimited-OCR-Q4_K_M.gguf" "mmproj-Unlimited-OCR-F16.gguf" --local-dir ./uocr

Run it on an image:

./build/bin/llama-mtmd-cli \
  -m ./uocr/Unlimited-OCR-Q4_K_M.gguf \
  --mmproj ./uocr/mmproj-Unlimited-OCR-F16.gguf \
  --image document.png \
  -p "<|grounding|>Convert the document to markdown." \
  --temp 0

Use --temp 0 for OCR (deterministic). Add -n 4096 (or more) for long/dense documents.

---

Prompting guide

Unlimited-OCR uses the DeepSeek-OCR prompt vocabulary. The prompt is just an instruction; prefix it with <|grounding|> whenever you also want bounding boxes for what was read.

Worked examples

1) Document → clean Markdown (tables, headings, reading order):

./build/bin/llama-mtmd-cli -m ./uocr/Unlimited-OCR-Q4_K_M.gguf \
  --mmproj ./uocr/mmproj-Unlimited-OCR-F16.gguf \
  --image invoice.png --temp 0 -n 4096 \
  -p "<|grounding|>Convert the document to markdown."

2) Just the raw text, no layout / no boxes:

./build/bin/llama-mtmd-cli -m ./uocr/Unlimited-OCR-Q4_K_M.gguf \
  --mmproj ./uocr/mmproj-Unlimited-OCR-F16.gguf \
  --image receipt.jpg --temp 0 -p "Free OCR."

3) Locate a specific string and get its box:

./build/bin/llama-mtmd-cli -m ./uocr/Unlimited-OCR-Q4_K_M.gguf \
  --mmproj ./uocr/mmproj-Unlimited-OCR-F16.gguf \
  --image form.png --temp 0 \
  -p "<|grounding|>Locate <|ref|>Invoice Number<|/ref|> in the image."

Understanding the output (grounding tokens)

With <|grounding|>, the model interleaves the recognized text with detection boxes:

<|det|>title [37, 64, 464, 132]<|/det|>INVOICE #2026-0623
<|det|>text  [37, 194, 350, 247]<|/det|>Bill To: Sahil Chachra
<|det|>text  [37, 483, 329, 543]<|/det|>Total Due: $44.00

Each [x1, y1, x2, y2] is the bounding box (top-left → bottom-right) of that span, in the coordinate space of the model's input image. Drop the <|det|>...<|/det|> tags if you only want the text, or parse them to overlay boxes / build a layout. Without <|grounding|> you get plain text (or Markdown) with no box tags.

Tip — long documents: Unlimited-OCR targets one-shot long-horizon parsing. For multi-page scans, run page-by-page and concatenate. If output ever repeats/loops on a dense page, add a mild repetition penalty, e.g. --repeat-penalty 1.05, and keep --temp 0.

---

Serving (OpenAI-compatible API)

./build/bin/llama-server \
  -m ./uocr/Unlimited-OCR-Q4_K_M.gguf \
  --mmproj ./uocr/mmproj-Unlimited-OCR-F16.gguf \
  -c 8192 --host 0.0.0.0 --port 8080

Call it with an image (base64 data URL):

IMG=$(base64 -w0 document.png)
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "temperature": 0,
  "messages": [{ "role": "user", "content": [
    { "type": "text", "text": "<|grounding|>Convert the document to markdown." },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,'"$IMG"'" } }
  ]}]
}'

Python (OpenAI SDK) is identical — point base_url at http://localhost:8080/v1, send a text part with the prompt above and an image_url part with the data URL.

About the model

• Architecture: DeepseekOCRForCausalLM — DeepEncoder vision (SAM-ViT-B + CLIP-L/14, 1024×1024 input, 16× downsample) → linear projector → DeepSeek-V2 MoE text decoder (12 layers, hidden 1280, 64 routed + 2 shared experts, 6 experts/token).
• Task: multilingual OCR / document parsing — single image, multi-page, and PDF (one-shot long-horizon parsing). The original supports gundam (crop) and base resolution modes.
• License: MIT (inherited from the base model).

How these were made

1. Converted baidu/Unlimited-OCR to GGUF with the PR #17400 convert_hf_to_gguf.py. The converter targets DeepSeek-OCR, so the config's top-level architectures was set to DeepseekOCRForCausalLM and language_config.architectures to DeepseekV2ForCausalLM (the model is otherwise byte-identical to DeepSeek-OCR's tensor layout).
2. Exported the text decoder (BF16) and the vision tower (--mmproj, F16) separately.
3. Built an importance matrix from a general-text corpus and produced the K-/i-quants with llama-quantize.
4. Verified: the BF16 GGUF + mmproj correctly OCR a test document (text + grounding boxes) via llama-mtmd-cli before quantizing.

Limitations

• Needs the PR #17400 llama.cpp build until DeepSeek-OCR support lands in main.
• Very low-bit i-quants (IQ3_XXS, IQ2_M) trade real accuracy for size — prefer Q4_K_M or higher for production OCR.
• The vision encoder runs in fp16 regardless of the chosen text quant.

Credits

• Base model: baidu/Unlimited-OCR (MIT) — builds on deepseek-ai/DeepSeek-OCR.
• GGUF / DeepSeek-OCR llama.cpp support: ggml-org/llama.cpp#17400.
• Quantized by sahilchachra.