T-RSI Fast Reflex OCR

A 1.58-bit ternary BitNet text-line recognizer in 461 KB, with a dependency-free C engine

[!WARNING] Experimental research release. Every number below was measured on one Apple Silicon machine against ground truth this project generated. It has not been independently replicated. Treat the figures as reproducible-by-you, not as verified.

Every weight is -1, 0 or +1, packed four to a byte. The model reads a 32-pixel-tall text strip and emits characters through CTC, decoded with a prefix beam search over a 27 KB character language model.

It ties or beats Tesseract 5 on character accuracy in every scenario measured, reads a line 8.8x faster in 5x less memory, and still returns fewer perfect lines. Full numbers and the reasons are below, including where it loses.

That speed figure is ARM-specific. It was measured on an Apple M4 Max, where the C engine uses NEON and the ARMv8.2 dot-product instruction. Those carry the engine: without any SIMD it runs at 144 ms/line, which is Tesseract's speed. See Hardware dependence before quoting a number.

The exact-line gap used to be 25.7 points and is now 3.6 to 6.6 depending on the sample. What closed it is multi-stretch fusion, described below: decode the same line at two stretches, align the frames, average the log-probabilities, and decode once. It replaced the hypothesis voting an earlier version of this card recommended, and is both more accurate and about 6x faster than it.

---

Measured results

Reproduce with python3 benchmark_vs_tesseract.py. Ground truth needs no hand labelling: a digital PDF already contains its own exact text, so rasterizing a page and cropping each line at the bbox the PDF reports gives pixels whose correct answer is known. Training and test documents are split by file and deduplicated by content fingerprint.

Character accuracy

On the larger 3,000-line corpus in the deep benchmark below, clean documents come out a tie rather than a win (97.72% against 97.62%). Two samples, two answers, both reported.

Exact-line match — where it loses

Tesseract's errors cluster onto a few hard lines and leave the rest clean. This model's scatter — single characters across many otherwise-good lines — and one stray character ruins a line as thoroughly as ten. That is the documented weakness of CTC, which treats every timestep as conditionally independent given the image and so cannot notice that the letters it just emitted do not form a word.

Multi-stretch fusion closes most of that gap, and is the default. Decode the same line at more than one horizontal stretch, align the frames with DTW, average the LOG-PROBABILITIES, and decode once. The stretches fail on different characters, so averaging lets them cancel.

Combining the numbers rather than the decoded strings is the point. An earlier version of this card recommended voting on decoded strings (ROVER); measured head to head on the same lines, aligning and averaging the log-probabilities beats it by 1.8 points of exact-line match, because a decoded string has already thrown away how sure the decoder was. Soft decoding beating hard is a Shannon-era result, and confusion-network combination beats ROVER in speech for the same reason.

C engine, 2,000 held-out lines, latency on an M4 Max:

Fusion costs one extra forward pass, no extra bytes, and no retraining. Alignment has to be DTW: assuming a stretch scales time uniformly and resampling linearly gives up 1.5 points, because the model does not fire on a uniformly scaled schedule.

By content type (real documents)

Alphanumeric identifiers are the weakest content for both engines. Random codes carry no language structure, so no language model can rescue a misread glyph.

---

Deep benchmark vs Tesseract 5

3,000 held-out lines from 8 unseen documents, three degradation levels. Run it yourself with python3 deep_benchmark.py.

5 wins, 2 losses, 1 tie.

Peak RAM rose from 6.1 MB when fusion became the default -- it holds a second set of frame log-probabilities. Throughput improved anyway, from 25.1 ms/line, because the hot paths were rewritten at the same time (see the C engine section).

Per document -- genuinely mixed

4 wins, 2 losses and 2 ties on character accuracy across the 8 documents. One file swings it either way; this is not a clean sweep.

Content type

Numbers and capitals are the weak spots, by 3.3 and 4.2 points. If your documents are mostly figures, this is the wrong tool.

Line length

It wins on long lines and loses on short ones -- and short lines have the best frame budget, so this is not the deletion problem that stretching solves.

This is the biggest unexplained loss in the model, and it survived a dedicated hunt. Tried and measured on short lines, none of which helped: turning the character LM off (it costs 2.7 points, not gains), priming the LM with a leading space, raising or lowering the insertion bonus, a wider beam, and padding the line with white margin so the first glyph is not at frame zero. The only thing that helps is more stretch, which fusion already does per line. Short lines are headers, captions and table cells set in larger type; why that hurts is not known.

---

C engine

csrc/trsi_ocr.c -- one file, no dependencies beyond libm. No BLAS, no image library; input is binary PGM parsed in twenty lines.

cd csrc && make
./trsi_ocr ../model_v4.trsi line.pgm          # fuses 2 stretches, the default
./trsi_ocr --fuse=full ../model_v4.trsi line.pgm    # 5 stretches, more accurate
./trsi_ocr --fuse=single ../model_v4.trsi line.pgm  # one decode, fastest
./trsi_ocr --self-check                       # verify the SIMD paths

It is a port of engine.py, not a generic BitNet runtime. bitnet.cpp (arXiv:2410.16144) provides ternary matmul kernels for LLMs but has no patch tokenizer, no CTC, no beam search and no character LM, which is most of what this model is.

Two measured notes from building it:

• The first version ran at 364 ms/line because it unpacked 2-bit codes inside the inner loop. Unpacking once at load costs 1.25 MB resident and made it 5.7x faster. At 6 MB, memory was no longer the constraint.
• NEON SIMD took it from 63.7 to 23.7 ms/line, bit-identical on all 300 test lines at the time.
• Profiling the fused engine put 97% of the time in two functions: the ternary matmul (50.7%) and attention (46.5%). The DTW alignment that fusion is named after was 0.6% -- fusion was never slow because of the fusion, only because it runs the transformer twice.
• Rewriting those two took it from 27.5 to 9.2 ms/line unfused and 52.0 to 17.5 ms fused, so the fused path now costs less than the unfused one used to. The matmul moved from float FMA to SDOT (int8 dot product, 16 multiply-accumulates per instruction) -- the old code widened ternary weights to float, doing multiplication on a format that exists to avoid it. Attention gained SIMD, a loop interchange that turned a stride-192 read of v into 32 contiguous floats, and a 4-wide polynomial exp in place of ~1.3 million libm expf calls per line.
• That last change gives up bit-identity by design, so it is bounded instead: ./trsi_ocr --self-check asserts the fast exp within 1e-5 relative of libm (measured 4.97e-6) and monotone, and the SDOT product within 1e-5 of a scalar reference (1.02e-6). End to end, exact-line match on 2,000 held-out lines moved by at most 0.30 points. Scalar fallbacks are kept for CPUs without the instructions.
• Measured and rejected: -O3, -funroll-loops and -march=native are all within noise of plain -O2 here (17.4-17.9 ms/line). Threading was not attempted -- a line recognizer is normally called on many lines, so the parallelism belongs in batching one level up.

model_v4.trsi (410 KB) is a flat binary holding the packed weights, the character LM and the decode parameters, in the exact order the C reads them -- no JSON parser, no name lookup.

Hardware dependence

Every throughput figure on this card is from one machine, an Apple M4 Max, and the speed is mostly the SIMD, not the C. The ternary matmul and attention are 97% of the runtime, so how well they vectorise decides everything.

Three build paths, same source, measured on that machine by disabling the higher ones. Fused two-stretch decode, 100 held-out lines, median of three runs:

Two things worth reading off that table.

The bottom row is the honest floor. With nothing vectorised the engine runs at 144 ms/line, which is Tesseract's 148 — the 8.8x advantage is an argument about SIMD, not about being small. The memory and install-size wins are unaffected.

Hand-written intrinsics are worth less than they look (16.6 against 20.7), because clang at -O2 on arm64 already vectorises the plain loops. The intrinsics buy the SDOT path, which the compiler will not choose on its own.

x86: it works, and it is leaving the main optimisation on the table.

There is no x86 machine in this evaluation, so there is no native x86 timing on this card and none should be inferred. What could be established on ARM:

• An -arch x86_64 build compiles, passes --self-check, and decodes 100 real lines with one line differing from the ARM build -- the same float-noise level that separates any two paths here. It is functionally equivalent.
• clang auto-vectorises the ternary matmul on x86, but as floats: mulps at the x86-64 baseline, vmulps with -mavx2. It emits no integer dot-product instruction. So x86 runs the arithmetic this engine used before the SDOT rewrite -- widening ternary weights to float to multiply them.
• x86's direct equivalent to SDOT is vpdpbusd (AVX-VNNI on Alder Lake and later, AVX512-VNNI on Ice Lake and Zen 4). The compiler here accepts it, so the instruction is reachable; it is simply not implemented in this source.

That is the concrete gap, and it is not written blind on purpose: a VNNI binary does not execute under Rosetta on this machine, so an x86 integer path could not be checked for correctness here, never mind timed. It needs real x86 hardware.

Rosetta timings are not reported. Running the x86 binary under translation measures the translator, not an x86 CPU.

The paths agree on text, not on bits. Across 100 lines the non-vectorised build produced identical output to NEON+SDOT, and the auto-vectorised one differed on a single line -- float reassociation, the same order of noise that separates the C and Python engines. --self-check prints which path a binary was compiled with.

---

Size, memory and speed

109x smaller on disk, 8.8x faster per line, 4.5x faster to cold start, and 5x less memory — in the C engine, on ARM with NEON and SDOT. Size and memory hold everywhere; the speed does not. Without SIMD the same code runs at 144 ms/line, level with Tesseract — see Hardware dependence.

In Python it uses more memory than Tesseract, 55.6 MB against 35.3 MB. The model is a small part of that: numpy and pillow cost 32.1 MB before a single weight loads. An earlier version of this card said a C port "would remove that, but no such port has been measured, so it is not a claim." It is now measured, and it is the row above.

Position encoding is fixed sinusoidal, computed at load and stored as zero bytes. A learned embedding at 1024×192 float32 would cost 768 KB — more than twice all the ternary weights combined.

---

Architecture

Inference is engine.py, NumPy only, no framework. Weights are ternary, so an integer SIMD or microcontroller port needs only additions and subtractions. On a CPU the same arithmetic runs through BLAS, which is not literally multiplication-free — the property belongs to the weights, not to this runtime.

engine.py is checked against the PyTorch training module on shared weights and must agree to float32 precision. That test exists because the two once silently computed different functions.

---

Quickstart

from PIL import Image
from engine import FastReflexOCREngine

engine = FastReflexOCREngine.from_pretrained(
    "psikosen/t-rsi-ocr",
    lm_name="char_lm_3gram.npz",   # omit for greedy CTC
    beam_width=4,
)

text, latency_us, ram_kb = engine.recognize(Image.open("line.png"))
print(f"{text!r}  ({latency_us:.0f} us)")

Two-stretch fusion is on by default. To trade accuracy against latency:

import fusion

engine.fuse = fusion.FULL     # 5 stretches: +2.75 exact-line points, 2.4x slower
engine.fuse = None            # single decode: 2x faster, -4.55 points

When a line must come back verbatim, ask for candidates instead of one answer. The beam already ranks its hypotheses, so the list is free, and the correct string is in the top 8 for 41% of the lines the top answer gets wrong:

from beam_decoder import prefix_beam_search

candidates = prefix_beam_search(
    logits, engine.blank_id, {i: t for i, t in engine.id_to_token.items() if len(t) == 1},
    lm=engine.lm, beam_width=8, n_best=3,
)

That is not a better recognizer, it is a different contract — useful when a downstream parser knows the shape it expects.

---

Two things that matter more than the model

1. Most documents need no OCR at all. DOCX files and digital PDFs already contain their exact text. Reading it takes 4.8 ms per page against Tesseract's 2,038 ms for the same page — 425× faster and exact rather than 97% accurate. Rasterizing a text layer and guessing it back loses table structure and turns 1999.00 into 999.00. Check for a text layer before reaching for any recognizer.

2. This is a line recognizer, not a page reader. It needs pre-cropped lines. Tesseract does layout analysis and recognition in one pass; this does not, and no line detector ships with it. A per-line speed comparison assumes the crops already exist.

---

Training

python3 train_v2.py --epochs 70 --pool 50000 --real 25000 \
    --layers 4 --hidden 192 --heads 6 --guide-weight 0.5 --out model_v4

49,022 synthetic lines across 111 screened system fonts, mixed with 19,816 real lines cropped from PDFs. Both matter: a synthetic-only model scored 81% on held-out fonts and 16% on real documents — it had learned the renderer.

Augmentation covers blur, sensor speckle, non-uniform lighting, ±5° skew, exposure drift, and horizontal scale jitter (1.0–1.4×). That last one is what makes input_stretch work at inference: CTC cannot emit more characters than it has frames, real lines average 1.29 frames per character, and stretching to 1.69 is worth +14.9 points of exact-line match. Applied to a model trained without the jitter, the same trick is a net loss.

Training uses an attention decoder as an auxiliary teacher (GTC, arXiv:2002.01276), discarded at export — it costs zero bytes and zero milliseconds at inference.

---

Limitations

• Line recognizer only. No page layout, no line segmentation, no table structure.
• 95 printable ASCII characters. No accents, no CJK, no mathematical notation, no typographic quotes.
• Exact-line match still trails Tesseract, 78.9% against 85.5% on the 3,000-line corpus with fusion on. It was 74.6%. Ask for an n-best list if a line must come back verbatim.
• Short lines are the weakest bucket and the least explained. Under 30 characters: 92.40% against Tesseract's 93.63%. They have the best CTC frame budget in the corpus and still the worst accuracy, so the usual explanation does not apply. Six fixes were measured and none helped.
• The Python engine uses more RAM than Tesseract (55.6 MB against 35.3 MB), because of numpy and pillow rather than the model. The C engine uses 7.1 MB.
• Alphanumeric identifiers are weak (83.7%), as they are for every engine measured.
• Fusion runs the model twice. All headline accuracy assumes it is on. --fuse=single is roughly 2x faster and gives up 4.55 exact-line points.
• The speed advantage is SIMD, and it is measured on one ARM machine. With no vectorisation the engine runs at 144 ms/line against Tesseract's 148 — a tie, not an 8.8x win.
• x86 runs correctly but without the main optimisation. The build is verified equivalent; clang vectorises it as floats and emits no integer dot product, so x86 gets the pre-SDOT arithmetic. A vpdpbusd path is the obvious fix and is not implemented. No native x86 timing exists for this engine.
• Evaluated on one machine, an Apple M4 Max. Independent replication welcome, and especially welcome on non-ARM hardware.

---

What was tried and did not work

Recorded so it is not rebuilt from the same reasoning.

---

Team C · experimental research release · v3 (fused decode, SIMD C engine)