AraBART Arabic OCR span infiller

Repairs spans that have been marked as OCR errors. Stage B of a two-model post-OCR correction pipeline.

Input format

The model expects flagged spans fenced, not deleted:

input   سنن <err> این </err> ماجه - حدثنا <err> علي بنمحمد </err>
output  سنن ابن ماجه - حدثنا علي بن محمد

Consecutive flagged words become one span, so the model is free to emit a different number of words than it was given — which is what allows بنمحمد → بن محمد.

Usage

Intended to be used through arabic-postocr, which handles marking, generation and splicing:

from arabic_postocr import Corrector

corrector = Corrector.from_pretrained()
print(corrector.correct("سنن این ماجه").text)

Do not run it unsupervised over a whole document

This model is only safe inside the splice. Its output is trusted only at positions the detector flagged; everywhere else the original text is copied through byte for byte. Without that bound, a whole-document corrector damages considerable amount of the text it touches. With it, damage is capped by the detector's false-alarm rate.

Limitations

• Cannot emit diacritics. AraBART's vocabulary is undiacritised — قَالَ رَسُولُ اللَّهِ tokenises to 14 tokens of which 7 are <unk>. Tashkil is preserved for unflagged words by the splice and lost inside repaired spans.
• Numerals and proper nouns are not reconstructable. Page numbers, folio references and rare names carry no context to predict from. The pipeline's gate exists partly to catch these and route them to a human.
• Trained on scanned Arabic books. Other domains untested.