LightOnOCR-3-0.8B (GGUF)

GGUF quantizations of lightonai/LightOnOCR-3-0.8B — the lightest LightOnOCR-3 model (Qwen3.5 vision-language architecture), with the same layout and visual understanding features as the 4B — optimized for local inference with llama.cpp and Ollama, with its vision projector. All quants use an importance matrix.

Quick Start

With Ollama

ollama run richardyoung/lightonocr-3-0.8b
# or straight from Hugging Face
ollama run hf.co/richardyoung/LightOnOCR-3-0.8B-GGUF:Q8_0

With llama.cpp

huggingface-cli download richardyoung/LightOnOCR-3-0.8B-GGUF \
    --include "*Q8_0*" "mmproj*" --local-dir ./models

./llama-mtmd-cli -m ./models/LightOnOCR-3-0.8B-Q8_0.gguf \
    --mmproj ./models/mmproj-LightOnOCR-3-0.8B-F16.gguf \
    --image page.png -p "grounding"

Prompts: send the image with an empty prompt for plain transcription (Markdown), or with the prompt grounding to get every block with a label and a 0–1000 bounding box, image descriptions and chart data as tables. LightOn notes that other instructions are out of distribution. Recommended sampling: temperature 0.2, top_p 0.9 (set in the Ollama tags).

Why This Model?

• Smallest and fastest LightOnOCR-3 model, under 1 GB
• Two modes: plain transcription, or grounding with labels and bounding boxes
• Visual understanding: image descriptions and chart data as tables
• Apache-2.0: research and commercial use

Use Cases

• High-volume OCR on CPU or small GPUs
• Edge and on-device document capture
• Layout-aware extraction

Available Quantizations

Quant Quality

Each quant compared with the F16 model using llama-perplexity --kl-divergence on the wikitext-2 test set (40 × 512-token chunks). KL divergence measures how far the quant's next-token probabilities drift from F16; lower is better. Top-token match is how often the quant picks the same next token as F16. As a rough guide, a KLD under 0.01 is hard to tell from F16, and from about 0.05 differences start to show in longer outputs. These numbers cover the text decoder only, on English prose. For OCR accuracy, see Verified below.

Verified

Tested through Ollama before upload (Q4_K_M and Q8_0) on a synthetic invoice with an empty prompt; Q4_K_M returned: 'INVOICE 4471 / Date: 2026-09-28 / Widgets x3 ...... $42.00 / TOTAL: $86.20'. Converted with --no-mtp: the upstream config declares a multi-token-prediction layer whose weights are not in the release.

Also Available

• LightOnOCR-3-4B — most accurate
• LightOnOCR-3-1B — LightOnOCR-2 architecture, drop-in upgrade

Other Models by richardyoung

• New Heretic models (reproducible, listed in the Heretic Grimoire): LFM2.5-1.2B | Granite-4.2-3B | Granite-4.2-8B | Olmo-3-7B-Think | Phi-4-mini | DeepSeek-R1-0528-Qwen3-8B | Agents-A1-4B | FrogNano-4B | Muse-Glimmer-30B | Hemmingway-1
• Abliterated/Uncensored models: Qwen2.5-7B | Qwen3-14B | DeepSeek-R1-32B | Qwen3-8B
• OCR & Vision: MiniCPM-V-4.6 GGUF | dots.mocr GGUF | Unlimited-OCR GGUF | olmOCR GGUF | LightOnOCR-3-4B GGUF | WeVisDoc-4B GGUF | Xiaomi-OCR-0 GGUF
• MLX quantizations (Apple Silicon): Kimi-K2 series | olmOCR MLX
• Healthcare/Medical: Synthea 575K patients dataset | CardioEmbed
• Research: LLM Instruction-Following Evaluation (arxiv:2510.18892)

---

Built & maintained by Richard Young · DeepNeuro