TrOCR Small Handwritten (ONNX, INT8 decoder)

This repository contains an ONNX version of microsoft/trocr-small-handwritten, a Transformer-based OCR model for recognizing single-line handwritten text from images.

• The encoder is kept in FP32.
• The decoder (merged decoder + decoder_with_past) uses dynamic INT8 quantization.

Compared with the FP32 ONNX export, the model is about 28% smaller (246 MB → 178 MB) with no measurable change in CER/WER on the evaluation sample below.

Model details

How it was created

1. Exported the original model with optimum-cli export onnx (encoder + merged decoder).
2. Quantized decoder_model.onnx and decoder_with_past_model.onnx separately with onnxruntime.quantization.quantize_dynamic (per_channel=True, op_types_to_quantize=["MatMul"]).
3. Merged the two quantized decoders back into one file with optimum.onnx.graph_transformations.merge_decoders.

Note: quantizing the already merged decoder directly does not work, because the MatMul nodes inside the If subgraphs are skipped by quantize_dynamic.

Usage

pip install "optimum-onnx[onnxruntime]" transformers pillow

from PIL import Image
from optimum.onnxruntime import ORTModelForVision2Seq
from transformers import TrOCRProcessor

repo_id = "oguzhandurmaz/trocr-small-handwritten-onnx-int8"

processor = TrOCRProcessor.from_pretrained(repo_id)
model = ORTModelForVision2Seq.from_pretrained(repo_id)

image = Image.open("line.png").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values

generated_ids = model.generate(pixel_values)
text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(text)

Evaluation

Accuracy

300 random lines from the test split of Teklia/IAM-line (seed 42), greedy decoding, max_new_tokens=64. CER/WER were computed with jiwer, after removing the spaces before punctuation that the IAM transcriptions contain (e.g. word , → word,).

¹ Quantizing the encoder as well gave the smallest model, but increased CER by about 1.5 points, so it is not the model in this repository.

Differences of a few hundredths of a point are within noise for a 300-line sample.

Resource usage

30 lines, processed one at a time, CPU only (Kaggle notebook, 2 CPU threads). Each model was measured in a separate process. RAM figures include the Python/torch/transformers baseline of the measuring script (about 1.7 GB), so only the differences between rows are meaningful. "CPU ms/line" is the CPU time summed over all threads.

Compared with the FP32 ONNX export, the INT8 decoder reduces weight size (-28%), peak RAM (about -140 MB), latency (-9%) and CPU time (-10%), without a measurable accuracy change.

Compared with the original PyTorch model, it is faster per line (-28%) and smaller on disk, but uses about the same amount of RAM (+4%) and CPU time. Most of the RAM is the Python/ML library baseline rather than the model weights, so quantizing the weights has a limited effect on total memory.

These are single runs on a shared CPU with a small sample. Results depend on the CPU (for example, INT8 acceleration instructions) and on the thread settings, so please measure on your own hardware and data.

Intended use and limitations

• Designed for one line of handwritten text per image. For full pages, first split the page into lines with a separate text-line detection step.
• The base model was fine-tuned on the IAM handwriting database, so accuracy is best on English handwriting with similar style. Performance on other languages, scripts, or very different handwriting may be lower. The IAM test lines used for evaluation are in-domain for the base model.

Citation

Please cite the original TrOCR paper:

@misc{li2021trocr,
  title={TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models},
  author={Minghao Li and Tengchao Lv and Lei Cui and Yijuan Lu and Dinei Florencio and Cha Zhang and Zhoujun Li and Furu Wei},
  year={2021},
  eprint={2109.10282},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Acknowledgements

All credit for the model architecture and training goes to the original authors at Microsoft. This repository only provides an ONNX conversion with a quantized decoder.