shrew-ocr-preview-GGUF

GGUF conversions of shrew-ocr-preview for llama.cpp. The model converts one document page image per request into a single JSON object containing document metadata, a summary, RAG-ready semantic chunks, and figures/tables with bounding boxes and HTML — see the merged model card for the output schema and full input contract.

Converted from the v0.4 bf16 merged checkpoint with llama.cpp release b10453 (native Granite4Vision support), the same conversion as v0.2/v0.3. Only the language model is quantized; the vision tower is f16 in the mmproj — the same layout as the GPTQ variant. As of v0.4 the released weights are a full-weight fork of the Granite base (merged weights only: bf16, GPTQ-8bit, GGUF); no further standalone LoRA adapters will be published — see Lineage on the merged model card.

Files

SHA256SUMS is included; verify downloads against it.

Serving

llama-server splits -c evenly across parallel slots, and a slot with less than the full 32k context crashes on dense pages (exit 139). The rule: set -c to N × 32768 for -np N — full context per slot. This is a correctness requirement independent of VRAM.

# single slot — safe everywhere (~2.8 pages/min on a 12 GB RTX 3060)
llama-server -m shrew-ocr-preview-Q8_0.gguf \
  --mmproj shrew-ocr-preview-mmproj-f16.gguf \
  -ngl 99 -c 32768 -np 1 --host 0.0.0.0 --port 8080

# two slots on 12 GB — KV-cache quantization REQUIRED (f16 KV at 64k crashes under load);
# measured fidelity-neutral, ~2x throughput (~5.7 pages/min)
llama-server -m shrew-ocr-preview-Q8_0.gguf \
  --mmproj shrew-ocr-preview-mmproj-f16.gguf \
  -ngl 99 -c 65536 -np 2 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 0.0.0.0 --port 8080

Multi-slot validation (50-page gate, 12 GB RTX 3060, paired per-page fidelity vs single-slot): -np 2 + q8_0 KV scored first-pass 46/50, precision 0.939, paired fidelity delta 0.0000 — vs 44/50 / 0.935 single-slot. q8_0 KV quantization alone is also fidelity-neutral (paired delta 0.0000). Larger cards can scale -np further under the same -c = N × 32768 rule (unmeasured beyond N=2). -np 2 with f16 KV on 12 GB crashes under load; do not run it without the KV quantization flags.

Zero-setup alternative: the ocr-cuda image in the shrew-server repo (docker compose -f docker-compose.yml -f docker-compose.ocr.yml up -d --build) serves this model with the Q8_0 files pulled from this repo at build time. Env vars NP / CTX / KV_QUANT expose the same knobs (default NP=1, always safe).

The client contract is that of the merged model card (36-value section_type taxonomy — consumers validating that field must accept it): glyph-routed bucket preprocessing, the structured_extraction system prompt, and the v0.4 decoding recipe: temperature 0.4 with presence_penalty 0.3 (--temp 0.4 --presence-penalty 0.3 on llama-server, or per request), one un-enforced retry at the same settings on a failed page, then fallback. Point shrew-server at this endpoint, or follow the merged card's Usage section for direct integration.

Validation

v0.4 Q8_0 — 24 pages (4 documents across the corpus's page-count range) through the reference server (shrew-server 0.4.0) at its shipped defaults, llama-server b10453 single slot -c 32768 with --temp 0.4 --presence-penalty 0.3: 24/24 pages clean on the first pass, 0 retries, 0 failed pages, 244 s; the server's per-page decoding record shows the v0.4 recipe on every page. The serving envelope below (single slot always safe; two slots on 12 GB only with q8_0 KV) is carried over from v0.2/v0.3 and not re-measured for v0.4.

v0.3 Q8_0 — smoke through the reference server on an RTX 3060, single slot, -c 32768, 7.4 GB VRAM: 3/3 pages of a dense engineering handbook parsed clean on the first pass (7 chunks, no repetition aborts, no schema coercion), 54 s for the three pages; bucket routing and output identical in shape to the vLLM GPTQ-8bit serve of the same pages (single slot, -c 32768, RTX 3060, through the reference server). The serving envelope (single slot always safe; two slots on 12 GB only with q8_0 KV) is carried over from the v0.2 measurements on the same recipe and hardware; not re-measured for v0.3 or v0.4.

v0.2 history — Q8_0 gate on 50 pages vs the vLLM GPTQ-8bit variant: first-pass 44/50 vs 43/50, extraction precision 0.935 vs 0.937, paired table-fidelity delta median 0.000; f16 spot-check 9/10 pages ok (at -c 16384; the full 32k contract needs ≥16 GB VRAM).

Versions

Changelog

v0.4 (2026-09-28, tag v0.4) — reinforcement-learning stage merged onto an unpublished full-epoch merge of v0.3 (see Lineage), shipped with a new decoding recipe (sampled first pass at temperature 0.4 / presence 0.3, one un-enforced retry at the same settings, then fallback). Paired OHR-Bench retrieval vs the v0.3 weights on the same server: better overall (p = 2×10⁻¹⁹), driven entirely by reading order on dense Chinese newspapers (.476 → .754), with no evidence type worse; failed pages 1.00 % → 0.32 %. Weights, GPTQ-8bit and GGUF pushed in lockstep; no adapter (see Lineage). Requires shrew-server ≥ 0.4.0 (ships the recipe and reports which decoding rung produced each page). Buckets / pinpoints and the 36-value section_type contract unchanged from v0.3.

v0.3 (2026-09-14, tag v0.3) — generation E6. New LoRA (same r=256 recipe, 1 epoch / 1,343 steps, eval_loss 0.1408 vs 0.1421) trained on the up-weighted dense/broadsheet slices with the open section_type taxonomy; merged bf16, GPTQ-8bit, GGUF and adapter pushed in lockstep. Promoted on six pre-registered criteria (retrieval paired PASS, product gates PASS, regions PASS, parity TIE, contract screens PASS, image surface flat). Loops on OmniDocBench newspapers 66 % → 46 % in isolation; figure far-false-positives 156 → 51. The INT8 variant was gated by a 215-page product-path parity card against bf16 (all criteria met; text similarity median 1.000 on pages both arms parsed). Requires shrew-server ≥ 0.3.14 for the 36-value section_type contract. Buckets / pinpoints unchanged from v0.2. Retrieval tables are now the exact-kNN read.

v0.2 — glyph-routed tile buckets (E4); first GPTQ-8bit and GGUF releases.

This is a preview: weights update in place under these names as the model improves, and the repos are pushed in lockstep from the same training generation. Each push's commit message records the generation — pin a commit (revision=) for reproducibility.

Base model: ibm-granite/granite-vision-4.1-4b (Apache 2.0). The released weights are a full-weight fork of the base (see Lineage): the language model and the upper vision-encoder attention layers and projectors are fine-tuned; the rest of the vision tower is unchanged.