Falcon-OCR v1.5: kernel-ready CPU files

Pre-packed weights of tiiuae/Falcon-OCR v1.5 (revision fe757d59) for the Rust CPU runner at github.com/amazingvince/falcon-ocr-cpu, plus a trained draft head that speeds up decoding.

Each model file holds every tensor in the layout the runner's kernels read and is memory-mapped in place. For the recorded INT16/INT8 comparison, loading takes about 10 ms against about 2 s to quantize the original FP32 checkpoint at start-up. These are not transformers checkpoints; use them with the runner above.

Files

Every model file also carries the FP32 embedding, vocabulary head, norms, projector and attention sinks, and an INT8 screen of the vocabulary head with its error bounds, which the runner uses to pick exactly the same greedy token as the full FP32 head. Their safetensors metadata records the source checkpoint hash, the recipe and a digest of every tensor (--verify-model-file checks it).

The filenames describe body weight storage: fp32, int16-g64, and int8-gptq-g64. INT16 uses scaled integer codes, not FP16/BF16. g64 means one FP32 scale per 64 body inputs; GPTQ INT8 is this runner's quantizer, not a GGUF encoding. Other model weights remain FP32, with an INT8 vocabulary-screen auxiliary.

The FP32 file uses falcon-ocr-kernel-fp32-v1; the INT16/INT8 files keep falcon-ocr-kernel-v1. Their metadata records the recipe and source checkpoint. The draft head has its separate format and role.

Upgrade the runner and download these files together. INT16 and INT8 replace the old near-exact and fast filenames, while --mode exact, --mode near-exact, and --mode fast keep their meanings. Older binaries need explicit --model-file paths for the renamed quantized files and cannot load the new FP32 format.

Usage

git clone https://github.com/amazingvince/falcon-ocr-cpu && cd falcon-ocr-cpu
cargo build --release --locked
python -m pip install huggingface_hub
hf download BEE-spoke-data/falcon-ocr-v1.5-cpu --local-dir models/falcon-ocr-cpu
target/release/falcon-ocr --model models/falcon-ocr-cpu run page.png --text              # near-exact
target/release/falcon-ocr --model models/falcon-ocr-cpu --mode fast run page.png --text  # fast
target/release/falcon-ocr --model models/falcon-ocr-cpu --mode exact run page.png --text # FP32
target/release/falcon-ocr --model models/falcon-ocr-cpu doctor --text                    # what it will do

Build prerequisites: Rust 1.94.0, a C/C++ compiler, CMake, and NASM on x86; see the runner README for platform-specific setup. The download command requires the Hugging Face Hub package installed above.

CPU targets are x86-64 with AVX2 or AVX-512 and aarch64 (NEON), with a scalar fallback. x86-64 has model-backed measurements; aarch64 has compilation and kernel-test coverage, without a model-backed run.

Accuracy

Against the FP32 model, teacher-forced along its outputs on 55 calibration pages (24,262 tokens):

On 118 fresh English OmniDocBench pages (figures the ground truth leaves out masked on every system's input), near-exact produced exactly FP32's tokens on every page. Fast mode matched FP32 token for token on 93 pages, and its character error rate against the ground truth was 0.11 points lower than FP32's overall, with no repetition loops: no measurable loss on printed English. On handwriting and degraded scans in the runner's 200-page held-out set, fast mode's 8-bit weights tipped some uncertain pages into repetition loops (the runner's repetition stop cuts them short); use near-exact for such pages. Details: docs/MODES.md.

Speed

Recorded on Ryzen 9 7950X (16 cores, DDR5), Windows 11, default flags, without the draft head. These are mode measurements, not new benchmarks of the repacked files:

With the draft head (used automatically when it is in the model folder), 16 English held-out pages, total time with the n-gram drafter → with the head, tokens identical:

--max-dimension auto (opt-in) picks 768, 1024 or 1536 pixels per page from image statistics and saves about a quarter of the CPU time on its development set. It changes the output and failed its held-out quality gate. A later figure-masked comparison was diagnostic and did not change that verdict (details).

The draft head

An EAGLE-3-style drafter: one decoder block over the features of the target's layers 2, 11 and 19 (text only), a 16,383-token draft vocabulary, trained first on next-token prediction and then on seven-step chains (weights 0.5^j). It proposes up to four tokens per step while its confidence is at least 0.35, and the model verifies them in one pass, so it never changes the output, only the speed. The runner uses it automatically when this file sits next to the model files and speculation is on (--drafter ngram or --speculate 0 turn it off).

Training data: about 45,000 English document pages, rendered as images and transcribed by the official Falcon-OCR vLLM image (greedy), after removing duplicates and every page that shares an image or a 13-gram of text with OmniDocBench. No non-commercial or evaluation-only data set was used. Sources, with the pages of the final training stage:

The pipeline, the recipe and every measurement are in research/draft-head.

Rebuilding the model files

The FP32 file preserves the pinned checkpoint's parameter bytes and adds the precomputed vocabulary screen. INT16 and GPTQ INT8 are fresh repacks of the previously published files, with identical tensors, metadata values, profiles and v1 formats. The draft head is unchanged. The encoding names and FP32 loading support do not introduce new accuracy or speed claims.

To rebuild from the pinned FP32 checkpoint instead (which reruns GPTQ for fast mode), install the Python helper dependencies and restore the locked calibration pages as described in Development, then run:

python scripts/fetch_reference.py --output artifacts/model
cargo build --release --locked --bins
mkdir -p artifacts/repacked
target/release/falcon-ocr --mode exact pack --output artifacts/repacked/falcon-ocr-v1.5-fp32.safetensors
target/release/falcon-ocr --mode near-exact pack --output artifacts/repacked/falcon-ocr-v1.5-int16-g64.safetensors
BIN=target/release/falcon-ocr-eval bash tools/make_gptq_overlay.sh
target/release/falcon-ocr --mode fast pack --output artifacts/repacked/falcon-ocr-v1.5-int8-gptq-g64.safetensors

Choose a new output directory for each build; on Windows use the .exe binary paths. Detailed recipes and artifact requirements are in docs/DEVELOPMENT.md.

License and attribution

Derived from tiiuae/Falcon-OCR by the Technology Innovation Institute, released under the Apache License 2.0. The model files are re-encodings and quantizations of those weights, and the draft head was trained on that model's features and outputs; all are redistributed under the same license. They are not an official TII release. Cite the original work: Falcon OCR tech report. The draft head's training data remains under its sources' own terms (table above).