LightOnOCR-3-4B (GGUF)

GGUF quantizations of lightonai/LightOnOCR-3-4B — LightOn's most accurate LightOnOCR-3 model (Qwen3.5 vision-language architecture): end-to-end OCR with layout and visual understanding — optimized for local inference with llama.cpp and Ollama, with its vision projector. All quants use an importance matrix.

Quick Start

With Ollama

ollama run richardyoung/lightonocr-3-4b
# or straight from Hugging Face
ollama run hf.co/richardyoung/LightOnOCR-3-4B-GGUF:Q4_K_M

With llama.cpp

huggingface-cli download richardyoung/LightOnOCR-3-4B-GGUF \
    --include "*Q4_K_M*" "mmproj*" --local-dir ./models

./llama-mtmd-cli -m ./models/LightOnOCR-3-4B-Q4_K_M.gguf \
    --mmproj ./models/mmproj-LightOnOCR-3-4B-F16.gguf \
    --image page.png -p "grounding"

Prompts: send the image with an empty prompt for plain transcription (Markdown), or with the prompt grounding to get every block with a label and a 0–1000 bounding box, image descriptions and chart data as tables. LightOn notes that other instructions are out of distribution. Recommended sampling: temperature 0.2, top_p 0.9 (set in the Ollama tags).

Why This Model?

• Most accurate of the LightOnOCR-3 family — LightOn's recommendation for most OCR tasks
• Two modes: plain transcription, or grounding with labels and bounding boxes for every block
• Visual understanding: short descriptions for images, charts turned into tables of their data
• Tables, receipts, forms, multi-column layouts and math notation
• Apache-2.0: research and commercial use

Use Cases

• Digitizing scanned documents and PDFs locally
• Layout-aware extraction with bounding boxes
• Chart and table data extraction for RAG

Available Quantizations

Quant Quality

Each quant compared with the F16 model using llama-perplexity --kl-divergence on the wikitext-2 test set (40 × 512-token chunks). KL divergence measures how far the quant's next-token probabilities drift from F16; lower is better. Top-token match is how often the quant picks the same next token as F16. As a rough guide, a KLD under 0.01 is hard to tell from F16, and from about 0.05 differences start to show in longer outputs. These numbers cover the text decoder only, on English prose. For OCR accuracy, see Verified below.

Verified

Tested through Ollama before upload (Q4_K_M and Q8_0) on a synthetic invoice with an empty prompt; Q4_K_M returned: 'INVOICE 4471 / Date: 2026-09-28 / Widgets x3 ....... $42.00 / TOTAL: $86.20'. Converted with --no-mtp: the upstream config declares a multi-token-prediction layer whose weights are not in the release.

Also Available

• LightOnOCR-3-1B — LightOnOCR-2 architecture, drop-in upgrade
• LightOnOCR-3-0.8B — fastest

Other Models by richardyoung

• New Heretic models (reproducible, listed in the Heretic Grimoire): LFM2.5-1.2B | Granite-4.2-3B | Granite-4.2-8B | Olmo-3-7B-Think | Phi-4-mini | DeepSeek-R1-0528-Qwen3-8B | Agents-A1-4B | FrogNano-4B | Muse-Glimmer-30B | Hemmingway-1
• Abliterated/Uncensored models: Qwen2.5-7B | Qwen3-14B | DeepSeek-R1-32B | Qwen3-8B
• OCR & Vision: MiniCPM-V-4.6 GGUF | dots.mocr GGUF | Unlimited-OCR GGUF | olmOCR GGUF | LightOnOCR-3-4B GGUF | WeVisDoc-4B GGUF | Xiaomi-OCR-0 GGUF
• MLX quantizations (Apple Silicon): Kimi-K2 series | olmOCR MLX
• Healthcare/Medical: Synthea 575K patients dataset | CardioEmbed
• Research: LLM Instruction-Following Evaluation (arxiv:2510.18892)

---

Built & maintained by Richard Young · DeepNeuro