LaTeX_OCR_rec fine-tuned on AlFrauch/im2latex
PaddleOCR LaTeX_OCR_rec fine-tuned for printed formula image → LaTeX recognition on a 200,000-sample
subset of AlFrauch/im2latex.
Architecture, tokenizer, preprocessing and decoding are identical to PaddleOCR v3.7.0; only the weights changed.
Model
| Architecture | LaTeX-OCR: HybridTransformer encoder (ResNetV2 stem, layers [2, 3, 7] → 4 ViT blocks, dim 256, 8 heads, patch 16) + LaTeXOCRHead Transformer decoder (4 layers, dim 256, 8 heads, cross-attention, attention-on-attention, GLU feed-forward) |
| Parameters | 25.56 M |
| Base weights | official PaddleOCR rec_latex_ocr_train/best_accuracy.pdparams |
| Input | grayscale image, width ≤ 672 px, height ≤ 192 px, padded to multiples of 16; normalization mean 0.7931, std 0.1738 |
| Output | LaTeX string, at most 512 tokens including [BOS]/[EOS] |
| Tokenizer | PaddleOCR latex_ocr_tokenizer.json, byte-level BPE, 8,000 tokens (unmodified) |
| Decoding | official: top-k filter (threshold 0.9) + sampling at temperature 0.333, no KV cache |
Files
| File | Size | Description |
|---|---|---|
inference.json, inference.pdiparams, inference.yml |
101 MB | inference model exported with PaddleOCR tools/export_model.py |
best_accuracy.pdparams |
100 MB | training checkpoint (dygraph weights) for further fine-tuning with PaddleOCR |
latex_ocr_tokenizer.json |
24 KB | tokenizer used for training and decoding |
config.yaml |
4 KB | full training configuration |
predict.py |
3 KB | standalone inference script (Paddle Inference) |
Usage
pip install paddlepaddle-gpu==3.3.1 -i https://www.paddlepaddle.org.cn/packages/stable/cu129/
pip install albumentations==1.4.24 tokenizers pyyaml opencv-python
git clone --depth 1 --branch v3.7.0 https://github.com/PaddlePaddle/PaddleOCR.git
python predict.py --paddleocr ./PaddleOCR --image formula.png
Output (tab-separated):
formula.png \mathcal{E}\mathcal{B}(\Delta)\simeq\left(\frac{\Delta}{T}\right)^{1-p}
Use --cpu for CPU inference. The exported files follow the same format as
PaddlePaddle/LaTeX_OCR_rec
(Global.model_name: LaTeX_OCR_rec). Note that tools/infer/predict_rec.py in PaddleOCR v3.7.0 rejects
LaTeX_OCR_rec inference directories.
To continue training, set Global.pretrained_model in a PaddleOCR LaTeX_OCR_rec config to
best_accuracy (without extension).
Training data
| Source | AlFrauch/im2latex, revision 403adaf00649332f7065d33034da51f79356916a, 1,586,584 pairs |
| Filtering | 1,461,238 pairs kept. Rejected: 121,984 image larger than 672×192; 594 image width or height < 32 px; 2,636 label with characters outside the tokenizer vocabulary (mostly U+00A0); 130 label longer than 510 tokens; 2 empty label. No label was truncated. |
| Label normalization | leading/trailing whitespace stripped; LaTeX otherwise unchanged |
| Split | grouped by identical label text or identical image bytes (no shared label or image across splits): train 1,432,047 / validation 14,726 / test 14,465 |
| Used for this model | 200,000 pairs sampled uniformly from the train split (seed 0) |
Training procedure
| Framework | PaddlePaddle 3.3.1 (CUDA 12.9), PaddleOCR v3.7.0 tools/train.py |
| Hardware | 1× NVIDIA GeForce RTX 5060 Ti 16 GB |
| Epochs / steps | 2 / 7,120 |
| Training time | 42 min |
| Batching | each batch holds images of the same padded size and a similar label length (tiers ≤64, ≤128, ≤256, ≤512 tokens); batch size 26–64 (mean 56), gradient accumulation 1 |
| Optimizer | AdamW, β = (0.9, 0.999), weight decay 0.01, gradient clipping (global norm 1.0) |
| Learning rate | cosine decay from 5e-5, linear warmup over the first 1.25 % of an epoch |
| Precision | automatic mixed precision O1 (float16), dynamic loss scaling |
| Augmentation | official LatexTrainTransform (shift/scale/rotate, grid distortion, RGB shift, Gaussian noise, brightness/contrast, JPEG compression, binarization p = 0.04) |
| Seed | 42 |
| Final training loss | 0.082 |
Evaluation
Validation subset: 2,000 images sampled from the validation split (seed 0). Metrics as in
ppocr.metrics.rec_metric.LaTeXOCRMetric: prediction and reference are both decoded with LaTeXOCRDecode
(whitespace not between two letters is removed) before comparison.
- ExpRate: exact string match of the whole expression
- BLEU: mean sentence-level BLEU
- Edit distance: character-level normalized Levenshtein distance (lower is better)
| Model | Step | ExpRate | BLEU | Edit distance |
|---|---|---|---|---|
| this model | 2,000 | 0.4525 | 0.8419 | 0.1326 |
| this model | 4,000 | 0.4915 | 0.8648 | 0.1158 |
| this model (released) | 6,000 | 0.5085 | 0.8691 | 0.1115 |
A separate evaluation of the released checkpoint on the same 2,000 images (fp32, seed 42) gave ExpRate 0.5035, BLEU 0.8687, edit distance 0.1108; the difference comes from the sampling decoder. The full validation and test splits were not evaluated.
Limitations
- Inputs larger than 672×192 px are cropped to the ink bounding box and downscaled by
MinMaxResize; such images were excluded from training. - Trained on 14 % of the filtered training split for 2 epochs; validation accuracy was still improving at the end.
- The source dataset contains identical renderings with different LaTeX strings (e.g. with/without
\displaystyle,{\rm d}vs\mathrm{d}), which bounds exact-match accuracy. - Decoding samples from the output distribution, so repeated runs can produce slightly different strings.
- Characters outside the tokenizer vocabulary (non-ASCII) cannot be produced.
- Inference is autoregressive without KV cache; runtime grows quadratically with output length.
Checksums (SHA-256)
| File | SHA-256 |
|---|---|
inference.json |
e6a1a69107231c68483b5e029ebdfdea83cb17a97731fac67e62a3a580c8244c |
inference.pdiparams |
e307368424e6ff1e220d4d2aa88ef6608dcd3bc2a93d1e20ffbe1fbe48d787b5 |
inference.yml |
31e9f4fa65e903fff3f32ea6c2016a0ae84e69de34eb28f32c2764093d84f9cd |
best_accuracy.pdparams |
e4e5f43ff1f0160381aee1a8d966b9afdca768fc9e701fcda4b073375923d223 |
latex_ocr_tokenizer.json |
50aff4b7f753cc27388cde5c4fda7cc8e19c5904ca1d993cb332187a6d3eb060 |
config.yaml |
2c63748f2c0429ce9884ecc4cdb7ae593b2afc84693b6c7849f2fc34097ca384 |