Model Card for arabic-nougat-base-books-ocr

Arabic book-page OCR: a page image in, the page as text out, split into [HEADER], [BODY] and [FOOTNOTES] with diacritics kept. MohamedRashad/arabic-base-nougat (349M) fine-tuned on scanned tafsir pages and synthetic Shamela pages. Intermediate checkpoint: step 20,000 of 50,000 planned steps; training has not reached its end.

Both training datasets are plain grayscale, so from step 15,213 on the model is trained with synthetic colour augmentation (tinted paper, coloured text, dark pages, phone-photo look). Held-out character error rate (diacritics included, 16 pages per pool): 0.0092 on the original pages and 0.0110 on coloured variants of the same pages at step 20,000; the coloured variants were at 0.5999 before colour training (base model before any fine-tuning: 0.5715 on the original pages). The coloured pages are simulated, see Limitations.

Model Details

Model Description

VisionEncoderDecoderModel (Swin encoder, mBART decoder), input 896x672 through NougatProcessor. Output per page is [HEADER] ..., [BODY] ..., [FOOTNOTES] ... on separate lines (NFKC-normalised, Arabic diacritics kept). Weights are fp16, exactly as saved by the training run at step 20,000. Another checkpoint scored better on the original pages: step 15000 has eval CER 0.0090 against 0.0092 here; this repo holds the latest.

• Model type: Image-to-text encoder-decoder Transformer (Swin + mBART), 349M parameters
• Language(s) (NLP): Arabic (printed book pages)
• Finetuned from model: MohamedRashad/arabic-base-nougat

Direct Use

Transcribe single scanned, photographed or rendered pages of printed Arabic books (classical, with diacritics) into text, one page image at a time. Coloured pages (tinted paper, coloured headings and frames, dark or coloured backgrounds, photos) are supported through synthetic colour augmentation; see Limitations for how that was tested.

Out-of-Scope Use

Handwriting, tables, multi-column layouts, non-book documents, other scripts. Pages whose transcription would need more than 3200 tokens were removed from training and evaluation, so very dense pages are untested. Any use that relies on the text being exact without review.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from PIL import Image
from transformers import NougatProcessor, VisionEncoderDecoderModel

MODEL_ID = "masterofaudio2077/arabic-nougat-base-books-ocr"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32     # the weights are stored in fp16
processor = NougatProcessor.from_pretrained(MODEL_ID)
model = VisionEncoderDecoderModel.from_pretrained(MODEL_ID, dtype=dtype).to(device).eval()


def ocr(image, **generate_kwargs):
    """One book page (PIL image) -> '[HEADER] ...\n[BODY] ...\n[FOOTNOTES] ...'. Greedy decoding, up to 3,200 new tokens."""
    pixel_values = processor(image.convert("RGB"), return_tensors="pt").pixel_values.to(device, dtype)
    with torch.no_grad():
        out = model.generate(pixel_values=pixel_values, **generate_kwargs)
    return processor.tokenizer.batch_decode(out, skip_special_tokens=True)[0].strip()


print(ocr(Image.open("page.png")))

Training Details

Training Data

freococo/ocr_arabic_books (config tafsir_al_tabari, about 16.7k scanned pages) and freococo/synth_shamela_ocr_arabic_books (50 randomly chosen books, seed 0). Each batch is half Tabari, half synthetic. Both datasets are CC-BY-NC-ND and plain grayscale.

Training Procedure

Preprocessing [optional]

• Target text. Each page becomes [HEADER] ... / [BODY] ... / [FOOTNOTES] ... on separate lines, NFKC-normalised, with diacritics kept.
• Length filter. Pages whose target is longer than 3,200 tokens are dropped from every pool, so each label fits the decoder's 4,096 positions with its end token.
• Image input. 896x672 through NougatProcessor.
• Colour augmentation (from step 15,213). With probability 70% a training page is restyled by one of four recipes before the usual degradation (the labels do not change): paper (yellowed, grey or pastel paper, tinted ink, uneven light, stains), coloured text (coloured header and line bands, coloured frames), dark (light text on dark or saturated pages, coloured panels), photo (colour cast, shadows, vignette, sensor noise, slight perspective on a table-coloured background). The remaining pages stay as they are, so grayscale quality is kept.
• Other augmentation. Mild scan-like degradation (small rotation, blur, contrast and brightness, JPEG), and Nougat-style random replacement of 5% of the decoder input tokens while the labels stay clean (meant to reduce repetition loops).
• Data mixing. Each batch is half real Tabari scans and half synthetic Shamela pages. The stream is a pure function of the step number, so a resumed session continues on the exact next sample.
• Held out. The last 400 Tabari pages (an unseen section of the book) and 5 synthetic books that are not among the 50 training books.

Training Hyperparameters

• Training regime: - Procedure. Full fine-tune of MohamedRashad/arabic-base-nougat (all parameters) with a custom PyTorch loop, two DDP processes, one per GPU. Steps 0 to 15,213 without colour augmentation, then the same run resumed with it. Metrics, qualitative reports and checkpoints were logged to MLflow throughout.
• Precision: fp16 autocast over fp32 master weights, with a gradient scaler. Gradient checkpointing is on.
• Optimiser: Lion, betas (0.9, 0.99), lr 1.5e-05, weight decay 0.1 on weight matrices only (none on biases and norms).
• Schedule: 500 linear warm-up steps, then cosine decay to 10% of the peak (1.5e-06) at the planned 50,000 steps. Gradient clipping: 1.0.
• Loss: cross-entropy with label smoothing 0.05 over the whole tagged sequence.
• Batch: 2 pages per GPU x 2 GPUs x 2 accumulation steps = 8 pages per optimiser step. Seed: 0.
• Evaluation during training: every 1,000 steps on fixed held-out pages: teacher-forced loss on 64 pages per pool and greedy-decoded CER/WER on 16 pages per pool, each page decoded as it is and as a fixed coloured variant, plus a qualitative page-diff report.
• Checkpoints: every 5,000 steps. fp16 weights are kept per checkpoint; the fp32 resume state (model, Lion state, scaler, step) is overwritten at the same path. Sessions resume by MLflow run id.

Speeds, Sizes, Times [optional]

20,000 of 50,000 planned steps (160,000 page samples) on 2x NVIDIA T4 (Kaggle), over several sessions resumed from one MLflow run. Throughput since colour augmentation started: median 3.4 pages/s; about 9.9 GB per GPU at micro-batch 2. Weights: 349M parameters, fp16 safetensors. Wall-clock time: not recorded.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Fixed held-out pages, never trained on: the last 400 Tabari pages (an unseen section of the book) and pages of 5 synthetic books not among the 50 training books. Generation is evaluated on the first 16 pages of each pool; teacher-forced loss on 64 per pool. Every page is also decoded as a coloured variant: a fixed, seeded restyling, with the four recipes assigned to pairs of pages in turn, so original and coloured CER are measured on the same pages and are comparable across checkpoints.

Factors

Source (real scans vs synthetic pages), page field (header, body, footnotes), and colour recipe (tinted paper, coloured text, dark pages, photo look).

Metrics

CER / WER: Levenshtein distance over the whole tagged page string (characters / whitespace-separated words) divided by reference length, summed over pages. Diacritics removed: the same after stripping harakat from reference and prediction. Per-field CER after splitting on the tags. Overlong / empty rate: share of outputs longer than 1.5x reference + 20 characters / empty. Loss, perplexity, token accuracy: teacher-forced, without label smoothing. Coloured CER and the gap (coloured minus original CER on the same pages).

Results

All numbers below are the values stored by the training run in MLflow (run d0015d16e2784093beace9b12ebbfdea), at step 20,000 (periodic evaluation). Intermediate checkpoint: step 20,000 of 50,000 planned steps; training has not reached its end. "Before colour training" is the same run evaluated at step 15,213, when colour augmentation was switched on.

Original (grayscale) pages

By pool:

Coloured variants of the same held-out pages

Training curves:

Stored files in this repo:

• evaluation/metrics_step020000.json
• evaluation/qualitative_eval_step020000.html
• evaluation/qualitative_eval_table_step020000.json
• evaluation/training_curves.png

The qualitative HTML report shows page image, reference and prediction with a word-level diff (original pages and one page per colour recipe); download it and open it in a browser (the Hub shows HTML files as source).

Summary

Original pages: CER 0.0092, WER 0.0409 (diacritics included); CER without diacritics 0.0057; base model before fine-tuning: CER 0.5715. Coloured variants: CER 0.0110 (was 0.5999 before colour training). Overlong outputs: 0.00% original, 0.00% coloured.

Model Examination [optional]

Qualitative examples (full report: evaluation/qualitative_eval_step020000.html, all decoded pages with CER: evaluation/qualitative_eval_table_step020000.json):

Tabari, unseen pages: the median page by CER (16 pages decoded, CER 0.0130, worst 0.0561, best 0.0059)

Reference (first 400 characters):

[HEADER] 
[BODY] الموضعِ: الجنودُ.

ذكرُ مَن قال ذلك
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال: ثنى عمي، قال: ثنى أبي، عن أبيه، عن ابن عباسٍ: ﴿وَفِرْعَوْنَ ذِي الْأَوْتَادِ﴾. قال: الأوتادُ: الجنودُ الذين يشُدُّون له أمره، ويقالُ: كان فرعونُ يُوتِدُ في أيديهم وأرجلهم أوتادًا من حديدٍ، يُعَلِّقُهم بها (١).
وقال آخرون: بل قيل له ذلك؛ لأنه كان يُوتِدُ الناس بالأوتادِ.

ذكرُ مَن قال ذلك
حدَّثني محمدُ ب

Prediction (first 400 characters):

[HEADER] 
[BODY] الموضعِ: الجنودُ.

ذكرُ مَن قال ذلك
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال: ثنى عمي، قال: ثنى أبي، عن أبيه، عن ابن عباسٍ: ﴿وَفِرْعَوْنَ ذِي الْأَوْنَادِ﴾. قال: الأوتادُ: الجنودُ الذين يشُدُّون له أمرَه، ويقالُ: كان فرعونُ يُوتِدُ في أيديهم وأرجلِهم أوتادًا من حديدٍ، يُعَلِّقُهم بها (١).
وقال آخرون: بل قيل له ذلك؛ لأنه كان يُوتِدُ الناسَ بالأوتادِ.

ذكرُ مَن قال ذلك
حدَّثني محمد

Synthetic, unseen books: the median page by CER (16 pages decoded, CER 0.0037, worst 0.0121, best 0.0000)

Reference (first 400 characters):

[HEADER] - ٥١٩ -
[BODY] كَمَا أَن من عصى عليا فأغضبه جَازَ لَهُ تأديبه
وَفِي الصَّحِيح عَن ابْن مَسْعُود عَن النَّبِي صلى الله عليه وسلم قَالَ مَا مِنْكُم من أحد إِلَّا وَقد وكل بِهِ قرينه من الْجِنّ
قَالُوا وَأَنت يَا رَسُول الله قَالَ وَأَنا إِلَّا أَلا أَن الله أعانني عَلَيْهِ فَأسلم فَلَا يَأْمُرنِي إِلَّا بِخَير
وَفِي الصَّحِيح عَن عَائِشَة ﵂ عَن النَّبِي صلى الله عليه وسلم بِنَحْوِهِ
وَقَوله

Prediction (first 400 characters):

[HEADER] - ٥٩٩ -
[BODY] كَمَا أَن من عصى عليا فأغضبه جَازَ لَهُ تأديبه
وَفِي الصَّحِيح عَن ابْن مَسْعُود عَن النَّبِي صلى الله عليه وسلم قَالَ مَا مِنْكُم من أحد إِلَّا وَقد وكل بِهِ قرينه من الْجِنّ
قَالُوا وَأَنت يَا رَسُول الله قَالَ وَأَنا إِلَّا أَلا أَن الله أعانني عَلَيْهِ فَأسلم فَلَا يَأْمُرني إِلَّا بِخَير
وَفِي الصَّحِيح عَن عَائِشَة ﵂ عَن النَّبِي صلى الله عليه وسلم بِنَحْوِهِ
وَقَوله 

Coloured, tinted or yellowed paper: the median page by CER (8 pages decoded, CER 0.0079, worst 0.0162, best 0.0000)

Reference (first 400 characters):

[HEADER] 
[BODY] اللَّهُمَّ أنزل نصرك
قَالَ الْبَراء وَكُنَّا إِذا احمر الْبَأْس نتقي بِهِ وَكَانَ الشجاع منا الَّذِي يحاذيه يَعْنِي النَّبِي صلى الله عليه وسلم
وَلمُسلم من حَدِيث سَلمَة بن الْأَكْوَع قَالَ لما غشوا النَّبِي صلى الله عليه وسلم نزل ثمَّ قبض قَبْضَة من التُّرَاب واستقبل بِهِ وُجُوههم فَقَالَ شَاهَت الْوُجُوه فَمَا خلق الله مِنْهُم إنْسَانا إِلَّا مَلأ عَيْنَيْهِ تُرَابا بِتِلْكَ الق

Prediction (first 400 characters):

[HEADER] 
[BODY] اللَّهُمَّ انزل نصرك
قَالَ الْبَراء وَكُنَّا إِذا احمر الْبَأْس نتقي بِهِ وَكَانَ الشجاع منا الَّذِي يحاذيه يَعْنِي النَّبِي صلى الله عليه وسلم
وَلمُسلم من حَدِيث سَلمَة بن الْأَكْوَعِ قَالَ لما غشوا النَّبِي صلى الله عليه وسلم نزل ثمَّ قبض قَبْضَة من التُّرَاب واستقبل بِهِ وُجُوههم فَقَالَ شَاهَت الْؤُجُوه فَمَا خلق الله مِنْهُم إنْسَانا إِلَّا مَلأ عَيْنَيْهِ تُرَابا بِتِلْكَ ال

Coloured, coloured text, headings and frames: the median page by CER (8 pages decoded, CER 0.0092, worst 0.0260, best 0.0037)

Reference (first 400 characters):

[HEADER] 
[BODY] لذى النُّهى والعقل (١)
حدَّثني عليٌّ، قال: ثنا أبو صالحٍ، قال: ثني معاويةَ، عن عليٍّ، عن ابن عباسٍ في قوله: ﴿لِذِي حِجْر﴾. يقولُ: لأولى النُّهى.
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال: ثنى عمي، قال: ثنى أبي، عن أبيه، عن ابن عباسٍ: ﴿هَلْ فِي ذَلِكَ قَسَمٌ لِذِي حِجْرٍ﴾. قال: ذوى الحِجا والنُّهى والعقلِ.
حدَّثنا ابنُ حميدٍ، قال: ثنا مهرانُ، عن سفيانَ، عن قابوسَ بن أبي ظبيانَ، عن 

Prediction (first 400 characters):

[HEADER] 
[BODY] لذي النُّهى والعقلِ (١).
حدَّثني عليٌّ، قال: ثنا أبو صالحٍ، قال: ثني معاويةُ، عن عليٍّ، عن ابن عباسٍ في قولِه: ﴿لِذِي حِجْرٍ﴾. يقولُ: لأولى النُّهى.
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال: ثنى عمي، قال: ثنى أبي، عن أبيه، عن ابن عباسٍ: ﴿هَلْ فِي ذَلِكَ قَسَمٌ لِذِي حِجْرٍ﴾. قال: ذوى الحِجا والنُّهى والعقلِ.
حدَّثنا ابن حميدٍ، قال: ثنا مهرانُ، عن سفيانَ، عن قابوسَ بن أبي ظبيانَ، 

Coloured, dark or saturated pages: the median page by CER (8 pages decoded, CER 0.0109, worst 0.0363, best 0.0036)

Reference (first 400 characters):

[HEADER] 
[BODY] أَتَاكَ أبو لَيْلَى يجوبُ بهِ الدُّجَى ... دُجَى الليلِ جَوَّابُ الفلاةِ عَثَمْثَمُ (١)
يعنى بقولِه: يجوبُ: يدخلُ ويقطعُ.
وبنحو الذي قلنا في ذلك قال أهلُ التأويلِ.

ذكرُ مَن قال ذلك
حدَّثني عليٌّ، قال: ثنا أبو صالحٍ، قال: ثني معاويةُ، عن عليٍّ، عن ابن عباسٍ في قوله: ﴿وَثَمُودَ الَّذِينَ جَابُوا الصَّخْرَ بِالْوَادِ﴾. يقولُ: فخرَقوها (٢)
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال: ث

Prediction (first 400 characters):

[HEADER] 
[BODY] أَتَاكَ أبو لَيْلَى يجوبُ بهِ الدُّجَى ... دُجَى الليلِ جَوَّابُ الفلاةِ عَثَمْثَمُ (١)
يعني بقولِه: يجوبُ: يدخلُ ويقطعُ.
وبنحوِ الذي قلنا في ذلك قال أهلُ التأويلِ.
ذكرُ مَن قال ذلك
حدَّثني عليٌّ، قال: ثنا أبو صالحٍ، قال: ثنى معاويةُ، عن عليٍّ، عن ابنِ عباسٍ في قولِه: ﴿وَثَمُودَ الَّذِينَ جَابُوا الصَّخْرَ بِالْوَادِ﴾. يقولُ: فخرَقوها (٢).
حدَّثني محمدُ بنُ سعدٍ، قال: ثنى أبي، قال

Coloured, phone-photo look (colour cast, shadows, perspective): the median page by CER (8 pages decoded, CER 0.0123, worst 0.0187, best 0.0000)

Reference (first 400 characters):

[HEADER] - ٥١٩ -
[BODY] كَمَا أَن من عصى عليا فأغضبه جَازَ لَهُ تأديبه
وَفِي الصَّحِيح عَن ابْن مَسْعُود عَن النَّبِي صلى الله عليه وسلم قَالَ مَا مِنْكُم من أحد إِلَّا وَقد وكل بِهِ قرينه من الْجِنّ
قَالُوا وَأَنت يَا رَسُول الله قَالَ وَأَنا إِلَّا أَلا أَن الله أعانني عَلَيْهِ فَأسلم فَلَا يَأْمُرنِي إِلَّا بِخَير
وَفِي الصَّحِيح عَن عَائِشَة ﵂ عَن النَّبِي صلى الله عليه وسلم بِنَحْوِهِ
وَقَوله

Prediction (first 400 characters):

[HEADER] - ٥٧٩ -
[BODY] كَمَا أَن من عصى عليا فأغضبه جَازَ لَهُ تأديبه
وَفِي الصَّحِيح عَن ابْن مَسْعُود عَن النَّبِي صلى الله عليه وسلم قَالَ مَا مِنْكُم من أحد إِلاَّ وَقد وكل بِهِ قرينه من الْجِنّ
قَالُوا وَأَنت يَا رَسُول الله قَالَ وَأَنا إِلاَّ أَلا أَن الله أعانني عَلَيْهِ فَأسلم فَلَا يَأْمُرنِي إِلاَّ بِخَير
وَفِي الصَّحِيح عَن عَائِشَة ﵂ عَن النَّبِي صلى الله عليه وسلم بِنَحْوِهِ
وَقَوله

Model Architecture and Objective

Swin encoder, mBART decoder (4,096 positions, 50,000-token vocabulary), cross-entropy with label smoothing over the tagged page sequence.

Hardware

Inference: a GPU with 4 GB or more is comfortable; CPU works but is slow for ~1,500-token pages.

Software

Card built with transformers 5.22.1, torch 2.11.0+cu128. Training run: MLflow run d0015d16e2784093beace9b12ebbfdea, experiment arabic-book-ocr.

Citation

BibTeX:

Architecture:

@misc{blecher2023nougat,
  title={Nougat: Neural Optical Understanding for Academic Documents},
  author={Lukas Blecher and Guillem Cucurull and Thomas Scialom and Robert Stojnic},
  year={2023},
  eprint={2308.13418},
  archivePrefix={arXiv}
}