TrOCR-Sin-Handwritten

A Sinhala text-line recognizer for handwritten text, released with the SinFUND and SinOCR datasets (SriDoc/datasets).

Model

• TrOCR-style vision encoder-decoder (VisionEncoderDecoderModel).
• The encoder was initialized from facebook/deit-base-distilled-patch16-224 (ImageNet weights). The decoder was initialized from NLPC-UOM/SinBERT-large, a RoBERTa model pre-trained on Sinhala text. The cross-attention layers started from random weights.
• No OCR-pretrained TrOCR weights were used.
• Input: one pre-cropped text-line image, resized to 224 x 224.
• Decoding: beam search with 4 beams and a maximum length of 64 tokens.

Training data

This model was trained in two stages:

1. On the SinOCR printed training split. That model is SriDoc/TrOCR-Sin-Printed.
2. Then on the SinOCR handwritten training split (908 lines) for 100 epochs (2,900 steps), with a batch size of 32 and a learning rate of 5e-5.

Results

Character error rate (CER; lower is better) on the SinOCR test splits:

How the CER is computed:

• Characters are Unicode code points.
• The CER is pooled over the whole test split.

SinOCR has no separate validation split, so the test split was also the evaluation set during training. The lowest test CER occurred at the final checkpoint, so checkpoint selection did not change this result.

Limitations

• Errors remain common. A CER of about 0.53 means roughly one character error for every two characters, so the output needs human checking.
• Test writers were seen in training. The handwritten test split is not writer-disjoint: 72% of its lines come from writers who also appear in training. The CER is therefore probably optimistic for new writers.
• Do not use this model to measure OCR accuracy on the SinFUND test forms. 132 of its training lines are crops of those forms. The dataset repository lists them in sinocr_handwritten_provenance.csv.
• The model reads single, pre-cropped lines. It does not find text on a page.

Usage

from transformers import VisionEncoderDecoderModel, TrOCRProcessor, AutoTokenizer, ViTImageProcessor
from PIL import Image

model = VisionEncoderDecoderModel.from_pretrained("SriDoc/TrOCR-Sin-Handwritten")
tokenizer = AutoTokenizer.from_pretrained("NLPC-UOM/SinBERT-large")
image_processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
processor = TrOCRProcessor(image_processor=image_processor, tokenizer=tokenizer)

image = Image.open("line.png").convert("RGB")   # one cropped text line
pixel_values = processor(image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

Citation

@article{gunathilaka2025sinfund,
  title   = {SinFUND and SinOCR: Benchmarks for Sinhala Handwritten OCR and Template-Free Form Understanding},
  author  = {Gunathilaka, Kavishka and Hewagama, Danusha and Pushpakumara, Supul and Ambegoda, Thanuja D.},
  journal = {Research Square preprint},
  year    = {2025},
  doi     = {10.21203/rs.3.rs-6976719/v1}
}

Training log

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

The "Cer" and "Validation Loss" columns were measured on the SinOCR handwritten test split.

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.15.1