Model Card for arabic-nougat-base-books-ocr

Arabic book-page OCR: a page image in, the page as text out, split into [HEADER], [BODY] and [FOOTNOTES] with diacritics kept. MohamedRashad/arabic-base-nougat (349M) fine-tuned on scanned tafsir pages and synthetic Shamela pages. Intermediate checkpoint: step 20,000 of 50,000 planned steps; training has not reached its end.

Both training datasets are plain grayscale, so from step 15,213 on the model is trained with synthetic colour augmentation (tinted paper, coloured text, dark pages, phone-photo look). Held-out character error rate (diacritics included, 16 pages per pool): 0.0092 on the original pages and 0.0110 on coloured variants of the same pages at step 20,000; the coloured variants were at 0.5999 before colour training (base model before any fine-tuning: 0.5715 on the original pages). The coloured pages are simulated, see Limitations.

Model Details

Model Description

VisionEncoderDecoderModel (Swin encoder, mBART decoder), input 896x672 through NougatProcessor. Output per page is [HEADER] ..., [BODY] ..., [FOOTNOTES] ... on separate lines (NFKC-normalised, Arabic diacritics kept). Weights are fp16, exactly as saved by the training run at step 20,000. Another checkpoint scored better on the original pages: step 15000 has eval CER 0.0090 against 0.0092 here; this repo holds the latest.

Model type: Image-to-text encoder-decoder Transformer (Swin + mBART), 349M parameters
Language(s) (NLP): Arabic (printed book pages)
Finetuned from model: MohamedRashad/arabic-base-nougat

Direct Use

Transcribe single scanned, photographed or rendered pages of printed Arabic books (classical, with diacritics) into text, one page image at a time. Coloured pages (tinted paper, coloured headings and frames, dark or coloured backgrounds, photos) are supported through synthetic colour augmentation; see Limitations for how that was tested.

Out-of-Scope Use

Handwriting, tables, multi-column layouts, non-book documents, other scripts. Pages whose transcription would need more than 3200 tokens were removed from training and evaluation, so very dense pages are untested. Any use that relies on the text being exact without review.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from PIL import Image
from transformers import NougatProcessor, VisionEncoderDecoderModel

MODEL_ID = "masterofaudio2077/arabic-nougat-base-books-ocr"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32     # the weights are stored in fp16
processor = NougatProcessor.from_pretrained(MODEL_ID)
model = VisionEncoderDecoderModel.from_pretrained(MODEL_ID, dtype=dtype).to(device).eval()


def ocr(image, **generate_kwargs):
    """One book page (PIL image) -> '[HEADER] ...\n[BODY] ...\n[FOOTNOTES] ...'. Greedy decoding, up to 3,200 new tokens."""
    pixel_values = processor(image.convert("RGB"), return_tensors="pt").pixel_values.to(device, dtype)
    with torch.no_grad():
        out = model.generate(pixel_values=pixel_values, **generate_kwargs)
    return processor.tokenizer.batch_decode(out, skip_special_tokens=True)[0].strip()


print(ocr(Image.open("page.png")))

Training Details

Training Data

freococo/ocr_arabic_books (config tafsir_al_tabari, about 16.7k scanned pages) and freococo/synth_shamela_ocr_arabic_books (50 randomly chosen books, seed 0). Each batch is half Tabari, half synthetic. Both datasets are CC-BY-NC-ND and plain grayscale.

Training Procedure

Preprocessing [optional]

Target text. Each page becomes [HEADER] ... / [BODY] ... / [FOOTNOTES] ... on separate lines, NFKC-normalised, with diacritics kept.
Length filter. Pages whose target is longer than 3,200 tokens are dropped from every pool, so each label fits the decoder's 4,096 positions with its end token.
Image input. 896x672 through NougatProcessor.
Colour augmentation (from step 15,213). With probability 70% a training page is restyled by one of four recipes before the usual degradation (the labels do not change): paper (yellowed, grey or pastel paper, tinted ink, uneven light, stains), coloured text (coloured header and line bands, coloured frames), dark (light text on dark or saturated pages, coloured panels), photo (colour cast, shadows, vignette, sensor noise, slight perspective on a table-coloured background). The remaining pages stay as they are, so grayscale quality is kept.
Other augmentation. Mild scan-like degradation (small rotation, blur, contrast and brightness, JPEG), and Nougat-style random replacement of 5% of the decoder input tokens while the labels stay clean (meant to reduce repetition loops).
Data mixing. Each batch is half real Tabari scans and half synthetic Shamela pages. The stream is a pure function of the step number, so a resumed session continues on the exact next sample.
Held out. The last 400 Tabari pages (an unseen section of the book) and 5 synthetic books that are not among the 50 training books.

Training Hyperparameters

Training regime: - Procedure. Full fine-tune of MohamedRashad/arabic-base-nougat (all parameters) with a custom PyTorch loop, two DDP processes, one per GPU. Steps 0 to 15,213 without colour augmentation, then the same run resumed with it. Metrics, qualitative reports and checkpoints were logged to MLflow throughout.
Precision: fp16 autocast over fp32 master weights, with a gradient scaler. Gradient checkpointing is on.
Optimiser: Lion, betas (0.9, 0.99), lr 1.5e-05, weight decay 0.1 on weight matrices only (none on biases and norms).
Schedule: 500 linear warm-up steps, then cosine decay to 10% of the peak (1.5e-06) at the planned 50,000 steps. Gradient clipping: 1.0.
Loss: cross-entropy with label smoothing 0.05 over the whole tagged sequence.
Batch: 2 pages per GPU x 2 GPUs x 2 accumulation steps = 8 pages per optimiser step. Seed: 0.
Evaluation during training: every 1,000 steps on fixed held-out pages: teacher-forced loss on 64 pages per pool and greedy-decoded CER/WER on 16 pages per pool, each page decoded as it is and as a fixed coloured variant, plus a qualitative page-diff report.
Checkpoints: every 5,000 steps. fp16 weights are kept per checkpoint; the fp32 resume state (model, Lion state, scaler, step) is overwritten at the same path. Sessions resume by MLflow run id.

Speeds, Sizes, Times [optional]

20,000 of 50,000 planned steps (160,000 page samples) on 2x NVIDIA T4 (Kaggle), over several sessions resumed from one MLflow run. Throughput since colour augmentation started: median 3.4 pages/s; about 9.9 GB per GPU at micro-batch 2. Weights: 349M parameters, fp16 safetensors. Wall-clock time: not recorded.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Fixed held-out pages, never trained on: the last 400 Tabari pages (an unseen section of the book) and pages of 5 synthetic books not among the 50 training books. Generation is evaluated on the first 16 pages of each pool; teacher-forced loss on 64 per pool. Every page is also decoded as a coloured variant: a fixed, seeded restyling, with the four recipes assigned to pairs of pages in turn, so original and coloured CER are measured on the same pages and are comparable across checkpoints.

Factors

Source (real scans vs synthetic pages), page field (header, body, footnotes), and colour recipe (tinted paper, coloured text, dark pages, photo look).

Metrics

CER / WER: Levenshtein distance over the whole tagged page string (characters / whitespace-separated words) divided by reference length, summed over pages. Diacritics removed: the same after stripping harakat from reference and prediction. Per-field CER after splitting on the tags. Overlong / empty rate: share of outputs longer than 1.5x reference + 20 characters / empty. Loss, perplexity, token accuracy: teacher-forced, without label smoothing. Coloured CER and the gap (coloured minus original CER on the same pages).

Results

All numbers below are the values stored by the training run in MLflow (run d0015d16e2784093beace9b12ebbfdea), at step 20,000 (periodic evaluation). Intermediate checkpoint: step 20,000 of 50,000 planned steps; training has not reached its end. "Before colour training" is the same run evaluated at step 15,213, when colour augmentation was switched on.

Original (grayscale) pages

Original (grayscale) held-out pages; lower is better except token accuracy	Base model (step 0)	Before colour training (step 15,213)	Step 20,000
CER, with diacritics	0.5715	0.0092	0.0092
WER, with diacritics	0.9201	0.0429	0.0409
CER, diacritics removed	0.5594	0.0053	0.0057
WER, diacritics removed	0.8577	0.0196	0.0192
CER, [HEADER] field	1.0000	0.0352	0.0352
CER, [BODY] field	1.0000	0.0092	0.0082
CER, [FOOTNOTES] field	1.0000	0.0088	0.0140
Overlong outputs (more than 1.5x reference length + 20 chars)	18.75%	0.00%	0.00%
Empty outputs	0.00%	0.00%	0.00%
Held-out loss (nats/token, teacher-forced)	0.7663	0.0692	0.0678
Perplexity	2.152	1.072	1.070
Token accuracy (teacher-forced)	85.11%	99.44%	99.50%

By pool:

Pool	Metric	Base model (step 0)	Before colour training (step 15,213)	Step 20,000
All held-out pages	cer	0.5715	0.0092	0.0092
All held-out pages	wer	0.9201	0.0429	0.0409
All held-out pages	cer_nodiac	0.5594	0.0053	0.0057
Tabari, unseen pages	cer	0.5583	0.0130	0.0146
Tabari, unseen pages	wer	0.9432	0.0605	0.0619
Tabari, unseen pages	cer_nodiac	0.5726	0.0064	0.0081
Synthetic, unseen books	cer	0.5846	0.0053	0.0039
Synthetic, unseen books	wer	0.8959	0.0246	0.0190
Synthetic, unseen books	cer_nodiac	0.5459	0.0040	0.0033

Coloured variants of the same held-out pages

Coloured variants of the same held-out pages (CER, diacritics included)	Before colour training (step 15,213)	Step 20,000	Now: coloured minus original, same pages
Original (grayscale) pages, for reference	0.0092	0.0092	-
All coloured variants	0.5999	0.0110	0.0018
Tinted or yellowed paper	0.0206	0.0086	0.0014
Coloured text, headings and frames	0.0377	0.0108	0.0018
Dark or saturated pages	1.6846	0.0143	0.0005
Phone-photo look (colour cast, shadows, perspective)	0.7076	0.0106	0.0033

Training curves:

Stored files in this repo:

evaluation/metrics_step020000.json
evaluation/qualitative_eval_step020000.html
evaluation/qualitative_eval_table_step020000.json
evaluation/training_curves.png

The qualitative HTML report shows page image, reference and prediction with a word-level diff (original pages and one page per colour recipe); download it and open it in a browser (the Hub shows HTML files as source).

Summary

Original pages: CER 0.0092, WER 0.0409 (diacritics included); CER without diacritics 0.0057; base model before fine-tuning: CER 0.5715. Coloured variants: CER 0.0110 (was 0.5999 before colour training). Overlong outputs: 0.00% original, 0.00% coloured.

Model Examination [optional]

Qualitative examples (full report: evaluation/qualitative_eval_step020000.html, all decoded pages with CER: evaluation/qualitative_eval_table_step020000.json):

Tabari, unseen pages: the median page by CER (16 pages decoded, CER 0.0130, worst 0.0561, best 0.0059)

Reference (first 400 characters):

[HEADER] 
[BODY] الموضعِ: الجنودُ.

ذكرُ مَن قال ذلك
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال: ثنى عمي، قال: ثنى أبي، عن أبيه، عن ابن عباسٍ: ﴿وَفِرْعَوْنَ ذِي الْأَوْتَادِ﴾. قال: الأوتادُ: الجنودُ الذين يشُدُّون له أمره، ويقالُ: كان فرعونُ يُوتِدُ في أيديهم وأرجلهم أوتادًا من حديدٍ، يُعَلِّقُهم بها (١).
وقال آخرون: بل قيل له ذلك؛ لأنه كان يُوتِدُ الناس بالأوتادِ.

ذكرُ مَن قال ذلك
حدَّثني محمدُ ب