GGUF quantizations of PaddlePaddle/PaddleOCR-VL (the 1.6 release) — a ~0.9 B vision-language OCR model (NaViT encoder + ERNIE-4.5-0.3B) — for running document OCR locally with llama.cpp.
These are community quants. The original model and weights are © PaddlePaddle, released under Apache-2.0; this repo redistributes derivative (quantized) copies under the same license, with attribution. No model architecture or behavior was changed — only weight precision.
👉 Use PaddleOCR-VL-1.6-Q4_K_M.gguf (the recommended file) + the required mmproj.
# 1. download the recommended quant + the vision projector (required)
huggingface-cli download SanjeevSOLANKI/PaddleOCR-VL-1.6-GGUF \
PaddleOCR-VL-1.6-Q4_K_M.gguf PaddleOCR-VL-1.6-mmproj.gguf --local-dir .
# 2. serve it (8 GB GPU → ~1.6 GB VRAM, 8-way parallel)
llama-server -m PaddleOCR-VL-1.6-Q4_K_M.gguf \
--mmproj PaddleOCR-VL-1.6-mmproj.gguf -c 8192 -np 8 -ngl 99 --port 8080
Then POST a page image with the text prompt OCR: (OpenAI vision format) to http://localhost:8080/v1/chat/completions. Prefer higher-precision quants? Use Q5_K_M / Q6_K / Q8_0 instead — same mmproj. See Validation and the ⚠️ repetition-loop note before production use.
| File | Quant | Size | Notes |
|---|---|---|---|
PaddleOCR-VL-1.6-Q4_K_M.gguf |
Q4_K_M | 286 MB | recommended — fastest; reproduces F16's values on well-formed pages (see validation) |
PaddleOCR-VL-1.6-Q5_K_M.gguf |
Q5_K_M | 326 MB | slightly higher precision |
PaddleOCR-VL-1.6-Q6_K.gguf |
Q6_K | 367 MB | near-F16 |
PaddleOCR-VL-1.6-Q8_0.gguf |
Q8_0 | 475 MB | highest-precision quant |
PaddleOCR-VL-1.6-mmproj.gguf |
F16 | 841 MB | vision projector (required for all of the above) |
The F16 reference is not re-hosted here (it's identical to upstream) — get it from the official GGUF repo. It's used below only as the comparison baseline.
The usual quant metric (KL-divergence or perplexity vs F16 on a text corpus) does not apply to this model. PaddleOCR-VL is a vision OCR model — given plain text without an image, it's out-of-distribution: measured F16 perplexity on academic text was ~160 (a real text-LM is ~5–30), and in that near-random regime the comparison is meaningless (Q8_0 even scored a lower perplexity than F16). So we validated on the actual OCR task instead.
All five models (F16 + four quants) OCR'd the same four pages (title/authors, dense body text, equation+table, references) with identical, deterministic settings (-np 1, temperature 0). At -np 1 the model is fully deterministic — F16-vs-F16 = 100% identical — so any quant divergence is real, not run-to-run noise.
| Quant | Size | bpw | Table values correct | Text similarity to F16 |
|---|---|---|---|---|
| F16 | 892 MB | 16.0 | reference | — |
| Q8_0 | 475 MB | 8.5 | ✅ all | 73% |
| Q6_K | 367 MB | 6.6 | ✅ all | 68% |
| Q5_K_M | 326 MB | 5.8 | ✅ all | 70% |
| Q4_K_M | 286 MB | 5.1 | ✅ all | 66% |
51.3%, 46.3%, 37 / 80, 82.5%, …) is reproduced correctly by all of Q4–Q8 — verified directly, not assumed.⚠️ Known failure mode — repetition loops. On one of the four test pages (a highly repetitive template layout), the bare model degenerated into a runaway loop — repeating a line hundreds of times. Critically, this happened to F16 and several quants alike (F16 was the worst; Q8 didn't loop; Q4 did) — there is no precision ordering, so it is a model/inference limitation, not a quantization defect, and it occurred even with
repeat_penalty 1.15. See Limitations for mitigations. The "values correct" finding above applies to well-behaved pages; repetition-prone layouts can break for any precision including F16.
Sanity-grade, not a benchmark: 4 pages, one document, English, accuracy checked against F16 + targeted value spot-checks, not against human ground-truth labels (e.g. OmniDocBench). It establishes "Q4 ≈ F16 on values for well-formed pages" — it is not a multilingual or large-corpus accuracy claim, and (see warning above) it does not mean every page transcribes cleanly. For rigorous accuracy numbers, see the upstream model's OmniDocBench results.
| Config | Time | VRAM |
|---|---|---|
| F16, sequential | 118 s | ~1.9 GB |
| F16, 8-way parallel | 57 s | ~2.1 GB |
| Q4_K_M, 8-way parallel | 34 s | ~1.6 GB |
Q4 is ~1.7× faster than F16 and lighter, because decode (the bottleneck, ~89% of runtime) reads the weights for every generated token — 4-bit weights = 1/3 the memory traffic.
llama-server -m PaddleOCR-VL-1.6-Q4_K_M.gguf \
--mmproj PaddleOCR-VL-1.6-mmproj.gguf \
-c 8192 -np 8 -ngl 99 --port 8080
Then send a page image with the prompt OCR: (OpenAI vision format). On an 8 GB GPU this runs at ~1.6 GB VRAM and 8-way parallel.
Tips
OCR:; custom instructions ("convert to markdown…") make it worse."repeat_penalty": 1.15 is not always enough. Stronger mitigations: raise repeat_penalty (≈1.3) and/or enable DRY sampling (--dry-multiplier 0.8), cap max_tokens, and detect runaway output and retry/flag the page. For repetition-heavy documents, prefer the official PaddleOCR-VL pipeline (its layout stage avoids feeding whole repetitive pages to the VLM) or a model with a built-in no-repeat-ngram processor.llama-quantize (llama.cpp), static k-quants from the F16 GGUF. Imatrix was intentionally not used: at Q4–Q8 the values are preserved on the OCR task (see validation), and sub-4-bit IQ quants aren't useful for a 286 MB model whose constraint is decode speed, not size.
All quantization, validation, and benchmarks were run on a single machine. Numbers (speed/VRAM) are specific to this setup and will differ on yours.
| GPU | NVIDIA RTX A4000 Laptop, 8 GB, compute 8.6 (Ampere), driver 573.91 |
| CPU / RAM / OS | Intel i7-11850H · 64 GB · Windows 11 (build 26200) |
| Runtime | llama.cpp (turboquant fork, build 4595fff, MSVC 19.44) — GGUFs also run on any mainline llama.cpp with PaddleOCR-VL support |
| Base model | PaddlePaddle/PaddleOCR-VL 1.6, F16 GGUF + F16 mmproj |
| Quantizer | llama-quantize from the same build, F16 → k-quant |
| Validation | 4 pages of an English arXiv paper (title, body, equation+table, references); -np 1, temp 0, -c 8192, -ngl 99, prompt OCR:, repeat_penalty 1.15; deterministic F16 reference |
| Perf benchmark | 21-page paper; -np 8, -c 32768, page render 200 DPI, 8 concurrent requests |
| Date | 2026-06-25 |
Apache-2.0, inherited from the base model PaddlePaddle/PaddleOCR-VL. See that repo for the original LICENSE/NOTICE. "PaddleOCR" is a trademark of its owners; this is an unofficial community quantization and is not endorsed by PaddlePaddle.
These are unofficial, community-produced quantizations, provided "as is" without warranty of any kind, express or implied, including fitness for a particular purpose. Use at your own risk.