trocr-large-rukopys

A TrOCR-Large (VisionEncoderDecoderModel, ~558M params) line recognizer for handwritten Ukrainian document text, fine-tuned on the Rukopys dataset.

The training data included handwritten, printed, annotation and table region crops from Rukopys.

TL;DR

Intended use

Recognizing cropped line/region images from handwritten Ukrainian documents, downstream of a layout detector. This recognizer was tuned for robustness across all text region classes.

For better handwritten class CER (~0.075), see sibling model Hukyl/trocr-large-rukopys-hw.

How to use

Note: load the processor from this repo, not from microsoft/trocr-large-handwritten — the tokenizer here is an extended Cyrillic vocabulary (50336 vs stock 50265).

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-large-rukopys"
processor = TrOCRProcessor.from_pretrained(repo)          # load from THIS repo
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("region_crop.png").convert("RGB")        # one detected line/region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, max_new_tokens=256, num_beams=1)  # greedy
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Lineage & training curriculum

Rukopys gold-only fine-tuning directly from broad pretrain, with no silver/pseudo-labels. Full fine-tune, optimizer: AdamW via the 🤗 Seq2SeqTrainer, cosine LR decay, max_target_length 256, online augmentation, seed 42, single L40S.

1. Base — Kansallisarkisto/cyrillic-htr-model (Apache-2.0): microsoft/trocr-large-handwritten fine-tuned on ~30k historical Cyrillic rows by the Finnish National Archives; extended tokenizer (vocab 50336).
2. Mixed pretrain — Hukyl/trocr-large-uk-handwritten (6 ep, LR 4e-5): UkrHandwritten ×3 + Cyrillic Handwriting + part of pumb-ai/synthetic-cyrillic-large. The general Ukrainian/Cyrillic handwriting reader.
3. Rukopys gold fine-tune (22 ep, LR 2e-5, warmup 0.05, weight decay 0.01 + weight EMA): the human-labeled gold training data, with page-level validation split. Note: to incorporate as much of the Rukopys gold split as possible, this checkpoint had a near-zero validation set (~5 pages): 22,330 train / 105 val crops. The published weights are the EMA average of the training tail.

Weight EMA (why the published weights are an average)

At first, an original model with proper validation split (20%) was tested on Rukopys. After ablation on the held out test set (measured by Kaggle leaderboard), smaller validation splits produced better tests results. Therefore, to incorporate the gold-split data into training, we used the best-CER epoch from previous training, and added 2 more epochs (20 -> 22) to accommodate for more training data. The achieved EMA near-zero val model produced best results on the test split.

Hyperparameters (as launched)

Fine-tune ran 17,556 steps in ~2.5h.

Online data augmentation

Online augmentation was applied during training. Only training crops were augmented (no augmented validation was measured).

Each crop was transformed once per epoch by one geometric + one or two photometric operations at random. Each class received a separate augmentation profile that was selected to minimize the distribution shift.

These simulate scanner/paper variation, ink thinning/bleed, and natural handwriting deformation, widening the range of appearances beyond the raw training crops.

Results

Protocol: greedy decode, max_new_tokens=512, page-level gold validation split, 1000-sample draw, seed 42. Labels are metric-normalized (LaTeX → Unicode, table PSV canonicalized) so CER/WER reflect the scored character set.

Why the numbers come from a sibling

Since this checkpoint included almost all of the gold split, measuring CER on the tiny val split would result in meaningless metrics. Instead, we report CER on the model that used a 20% validation split, i.e. 185 pages. We suspect that this model is strictly equal or better than the reported numbers.

Per class

We also acknowledge that printed/table/annotation n is quite small, so measuring CER against them is quite noisy.

Limitations & biases

• Handwritten Ukrainian archival document material only; expect degradation on other scripts, languages, or modern born-digital text.
• The character set and normalization are tuned to the document-OCR metric — outputs are normalized text, not faithful transcription.
• table is weak and annotation/printed are high-variance (small n).
• Headline metrics are from a same-recipe sibling, not this exact checkpoint (its own holdout is too small to score); EMA weights are an average, not a single training step.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution