LightOnOCR-3-0.8B-cultural-heritage-eng
lightonai/LightOnOCR-3-0.8B fine-tuned to transcribe handwritten
English-language archival documents from Yale Library collections, following Smithsonian
Transcription Center style conventions.
Given a page image and the transcription prompt (shipped here as prompt.txt), the
model returns the page line by line, preserving original spelling, punctuation and capitalisation,
with editorial markup for deletions, marginalia, uncertain readings, tables, seals, stamps and
blank pages.
- Training data: 11,029 pages from
yale-cultural-heritage/ai-project-transcriptions - Transcriptions: prepared by Jon Manton and his team
- Evaluation: 582 held-out pages from 39 documents. No page from these documents was seen in training.
This is the fifth best of the models trained on this data (see the comparison below). The
best is yale-cultural-heritage/LightOnOCR-3-4B-cultural-heritage-eng.
Results
All numbers come from the same evaluation job, so the base and fine-tuned rows are a direct comparison. Decoding is greedy, with up to 2,048 new tokens and 2,048 visual tokens per page.
| metric (582 held-out pages) | base LightOnOCR-3-0.8B | this model |
|---|---|---|
| CER | 1.0221 | 0.1159 |
| WER | 1.0024 | 0.1320 |
| line recall | 0.0773 | 0.5696 |
| pages transcribed exactly | 0.0% | 14.4% |
| degenerate (looping) pages | 1.9% | 0.2% |
CER and WER are micro-averaged (total edits ÷ total reference characters or words), after Unicode NFD normalisation. Line recall is the share of reference lines reproduced exactly and in order.
The untuned LightOnOCR-3-0.8B scores above 1.0 because it does not follow the prompt's output conventions and often generates far more text than the page holds. CER counts every extra character as an error, so it can exceed 1.0.
How much of the error is markup?
Much of the remaining error is in the editorial markup, not in the reading. The reference and the prediction for the 527 non-blank held-out pages were rescored with markup removed from both sides:
| what is ignored | CER |
|---|---|
| nothing (whitespace tidied) | 0.1076 |
[[...]] tags ([[strikethrough]], [[left margin]], [[?]], [[seal]] ...) |
0.0560 |
+ abbreviation expansions, y[ou]r → yr |
0.0523 |
+ table column separators | |
0.0505 |
The double-bracket tags are under 5% of the reference text but account for roughly half of all character errors. The model uses the right kinds of tags, but it often places them differently from the human transcriber. Treat the tags as useful but approximate, and the underlying text as the reliable part.
Compared with other models trained on the same data
The same data and the same evaluation pages were used to train other open vision-language models, each with both LoRA and full fine-tuning (16 models in all). The seven best:
| model | tuning | CER | line recall |
|---|---|---|---|
| LightOnOCR-3-4B | LoRA | 0.0770 | 0.607 |
| Qwen3.5-4B | LoRA | 0.1047 | 0.590 |
| Qwen3.5-9B | LoRA | 0.1099 | 0.616 |
| chandra-ocr-2 | LoRA | 0.1101 | 0.596 |
| LightOnOCR-3-0.8B (this model) | LoRA | 0.1159 | 0.570 |
| Qwen3.5-2B | LoRA | 0.1190 | 0.573 |
| LightOnOCR-3-4B | full fine-tune | 0.1260 | 0.571 |
Differences under about 0.02 CER are within run-to-run noise in this project, so the models from Qwen3.5-4B to Qwen3.5-2B are close to tied. Smaller models are cheaper to run.
Usage
The model was trained with the prompt in prompt.txt, and it should be served with
that same prompt.
pip install "transformers>=5.5.4" torch pillow huggingface_hub
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
REPO = "yale-cultural-heritage/LightOnOCR-3-0.8B-cultural-heritage-eng"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
REPO, dtype=torch.bfloat16, device_map="auto"
).eval()
processor = AutoProcessor.from_pretrained(REPO) # carries the 2,048-visual-token page budget
prompt = open(hf_hub_download(REPO, "prompt.txt"), encoding="utf-8").read()
image = Image.open("page.jpg").convert("RGB")
messages = [{
"role": "user",
"content": [{"type": "image"}, {"type": "text", "text": prompt}],
}]
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = processor(text=[text], images=[[image]], return_tensors="pt").to(model.device)
with torch.inference_mode():
out = model.generate(
**inputs,
do_sample=False, # greedy, as evaluated
max_new_tokens=2048,
)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())
enable_thinking=False matters: without it some Qwen3.5 chat templates open a reasoning block,
and a model trained to transcribe directly then never reaches the transcription.
generation_config.json sets do_sample=False and repetition_penalty=1.1, and stops on
<|im_end|>, the end-of-turn token the model was trained to emit. The penalty is there for serving:
across these experiments it suppressed occasional repetition loops. The table above was produced
without it (repetition_penalty=1.0); pass that value to reproduce it exactly.
With vLLM
vllm serve yale-cultural-heritage/LightOnOCR-3-0.8B-cultural-heritage-eng \
--default-chat-template-kwargs '{"enable_thinking": false}'
import base64, requests
from huggingface_hub import hf_hub_download
REPO = "yale-cultural-heritage/LightOnOCR-3-0.8B-cultural-heritage-eng"
prompt = open(hf_hub_download(REPO, "prompt.txt"), encoding="utf-8").read()
image_b64 = base64.b64encode(open("page.jpg", "rb").read()).decode()
r = requests.post("http://localhost:8000/v1/chat/completions", json={
"model": REPO,
"messages": [{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
{"type": "text", "text": prompt},
]}],
"temperature": 0,
"max_tokens": 2048,
})
print(r.json()["choices"][0]["message"]["content"])
Do not put a literal <image> placeholder in the text when calling an OpenAI-style API. The image
is a separate content part, and a stray <image> string degrades output.
Training
The model was trained by supervised fine-tuning on page images paired with their reference
transcriptions. Its input was the page image followed by the prompt, and its target was the
transcription. Only the transcription tokens were scored in the loss; the prompt and image were
masked. A LoRA adapter was trained on both the vision encoder and the language decoder and
then merged into the base weights, so this repository is a standalone checkpoint with no peft
dependency.
The vision encoder is adapted as well because the input (handwritten archival documents) is unlike the printed documents LightOnOCR-3-0.8B was built for. In earlier experiments in this project, adapting the encoder was the single largest improvement measured.
| base model | lightonai/LightOnOCR-3-0.8B (Qwen3_5ForConditionalGeneration) |
| method | LoRA, merged after training |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
| adapted modules | 237 linear layers, vision tower and language decoder |
| trainable parameters | 27 M of 880 M (3.0%) |
| training pages | 11,029 (4 of 11,033 longer than the 8,192-token budget were dropped, not truncated) |
| epochs | 3 (4,137 optimizer steps) |
| effective batch | 8 (2 × 4 gradient accumulation) |
| learning rate | 1e-4, cosine schedule |
| max sequence length | 8,192 tokens |
| visual tokens per page | 2,048 (≈2.1 MP after resize) |
| precision | bf16, gradient checkpointing |
| hardware | 1 × NVIDIA RTX PRO 6000 Blackwell (96 GB), Yale Bouchet cluster |
| wall time | 8.6 h |
| validation loss by epoch | 0.234 / 0.212 / 0.218 |
Sequences are never truncated. A truncated target teaches the model to stop mid-page, and a truncated prompt removes the rules the target follows, so over-length samples are dropped instead.
The held-out split
The evaluation pages come from 39 whole documents (archival folders), stratified across the
collections. A page-level random split would put pages in the same hand on both sides, and would
measure memory of that hand rather than reading. The 39 documents and their 582 pages form the
validation split of the dataset.
Limitations
- English-language archival handwriting, Yale collections. The training pages come from five
collections:
ru_151,ycal_mss_1616,ru_1154,ru_0056andgen_mss_1457. Expect weaker results on hands, periods, languages and document types far from these. - Markup is a convention, not ground truth. Where to place
[[?]], which deletions to tag, and when a mark counts as a[[stamp]]or a[[seal]]are judgement calls. The model learned the conventions, but its tag placement often differs from a human's (see above). - Differences under about 0.02 CER are not meaningful. In this project, repeated runs of one configuration differ by up to 0.02 CER.
- The output needs review. About 57% of lines come out exactly right, so plan for human review before using it as an edition.
Acknowledgements
The transcriptions were prepared by Jon Manton and his team. The images are from Yale Library collections. The base model is LightOn's LightOnOCR-3-0.8B, released under Apache 2.0.