trocr-large-rukopys-hw

A TrOCR-Large (VisionEncoderDecoderModel, ~558M params) line recognizer for handwritten Ukrainian document text, fine-tuned on the Rukopys dataset.

The training data included handwritten, printed, annotation and table region crops from Rukopys.

TL;DR

Intended use

Recognizing cropped line/region images from handwritten Ukrainian documents, downstream of a layout detector. This recognizer was tuned for the strongest handwritten class reading.

For a recognizer more balanced across all region classes, see sibling model Hukyl/trocr-large-rukopys.

How to use

Note: load the processor from this repo, not from microsoft/trocr-large-handwritten — the tokenizer here is an extended Cyrillic vocabulary (50336 vs stock 50265).

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-large-rukopys-hw"
processor = TrOCRProcessor.from_pretrained(repo)          # load from THIS repo
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("region_crop.png").convert("RGB")        # one detected line/region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, max_new_tokens=256, num_beams=1)  # greedy
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Lineage & training curriculum

Rukopys gold-only fine-tuning from a real-only consolidation checkpoint, with no silver/pseudo-labels. Full fine-tune, optimizer: AdamW via the 🤗 Seq2SeqTrainer, cosine LR decay, max_target_length 256, online augmentation, seed 42, single L40S.

1. Base — Kansallisarkisto/cyrillic-htr-model (Apache-2.0): microsoft/trocr-large-handwritten fine-tuned on ~30k historical Cyrillic rows by the Finnish National Archives; extended tokenizer (vocab 50336).
2. Mixed pretrain — Hukyl/trocr-large-uk-handwritten (6 ep, LR 4e-5): UkrHandwritten ×3 + Cyrillic Handwriting + part of pumb-ai/synthetic-cyrillic-large.
3. Real-only consolidation — Hukyl/trocr-large-uk-handwritten-real (16 ep, LR 2e-5): UkrHandwritten ×2 + Cyrillic Handwriting, no synthetic data. The general Ukrainian handwriting reader.
4. Rukopys gold fine-tune (20 ep, LR 1e-5, warmup 0.05, weight decay 0.01): the human-labeled gold training data, with page-level validation split. Note: a smaller validation split was used to give more of the Rukopys gold split to training: 21,720 train / 1,223 val crops. The published weights are the best-val-CER epoch (10 of 20).

Hyperparameters (as launched)

Fine-tune ran 13,300 steps in ~2.5h.

Online data augmentation

Online augmentation was applied during training. Only training crops were augmented (no augmented validation was measured).

Each crop was transformed once per epoch by one geometric + one or two photometric operations at random. Each class received a separate augmentation profile that was selected to minimize the distribution shift.

These simulate scanner/paper variation, ink thinning/bleed, and natural handwriting deformation, widening the range of appearances beyond the raw training crops.

Results

Protocol: greedy decode, max_new_tokens=512, page-level gold validation split, 1000-sample draw, seed 42. Labels are metric-normalized (LaTeX → Unicode, table PSV canonicalized) so CER/WER reflect the scored character set.

Overall

A full-val (1,223-crop) training eval pass scores CER 0.0890 / WER 0.2661, consistent with the 1000-sample draw. The validation split is small (~67 pages), so the absolute numbers are somewhat noisy.

Per class

We also acknowledge that printed/table/annotation n is quite small, so measuring CER against them is quite noisy.

Limitations & biases

• Handwritten Ukrainian archival document material only; expect degradation on other scripts, languages, or modern born-digital text.
• The character set and normalization are tuned to the document-OCR metric — outputs are normalized text, not faithful transcription.
• table is weak and annotation/printed are high-variance (small n).
• Single seed and validation split — no across-run variance estimate.

Training data & attribution