Baberu OCR

English · 日本語

Baberu OCR reads the text inside a manga speech bubble. One 115M model, three languages. On JMangaBench_Read, a public benchmark of recent manga it never trained on, it is level with a 0.9B manga-tuned VLM and ahead of it on handwriting and sound effects; on Chinese and English it goes toe-to-toe with a general model 8× its size — all measured on independent ground truth.

v1.2 (2026-10): sound effects and handwriting. Up to v1.1 every Japanese label was manga-ocr's own output, so Baberu read about as well as its teacher, and nothing had ever shown it a sound effect. v1.2 adds the human labels of Manga109-s — 74 of its 87 titles, 105k dialogue boxes and 37k sound effects — and holds the other 13 out for evaluation. On JMangaBench_Read, sound effects go from 50% to 74% exactly right, handwriting from 60% to 67%, and the composite from 93.4 to 94.9 — see Benchmarks. All numbers on this page are v1.2's; --revision v1.1 and --revision v1.0 keep the earlier versions.

v1.1 (2026-10): long bubbles. v1.0 stopped reading at about 64 characters, so long English narration boxes came back cut off (discussion #2). v1.1 reads up to 256 — see Long bubbles.

Most manga OCR is Japanese-only and assumes clean horizontal lines. Real manga bubbles are not that. They run vertical, pack in sound effects, switch scripts mid-sentence, and mix full-width and half-width characters. Baberu was trained for exactly that, in Japanese, Chinese, and English at once.

At a glance

• Three languages, one checkpoint (ja / zh / en). On recent Japanese manga it is level with a 0.9B manga-tuned VLM and reads sound effects and handwriting better; on Chinese and English it matches a model 8× its size on the typical bubble, and is more robust.
• 115M parameters, about one-eighth the size of the 0.9B vision-language models it goes toe-to-toe with.
• Character-level (14,630-symbol vocab): sound effects, rare kanji, and full/half-width mixing stay one token each, with no subword fragmentation.
• Local and small. Apache-2.0, shipped as a 122 MB or 243 MB ONNX build that needs only onnxruntime, numpy, and pillow.

>>> from inference import BaberuOCR
>>> ocr = BaberuOCR(".")
>>> ocr(Image.open("bubble.jpg"))
'私がこの手で啓太を守る！！'

How it works

DINOv2 looks, a small decoder writes.

The decoder is built from scratch: 6 layers, grouped-query attention, SwiGLU, RMSNorm sandwich, 1D RoPE, tied embeddings. Nothing exotic. What mattered was the training order.

Freeze the vision encoder, train the decoder, then unfreeze the encoder and keep training. The accuracy jumps in that second stage, and it saturates fast: around 30k steps, roughly a tenth of an epoch. v1.2 continues from v1.1 with the encoder unfrozen for 12k more steps, mixing the Manga109-s human labels into the stream at 25%. Character-level output fits the task — the target is a sequence of characters — and avoids the uneven, ballooning tokenizations that subword vocabularies produce on sound effects and mixed scripts (manga-ocr is character-level for the same reason).

Benchmarks

Teacher labels would grade their own homework, so every number here is measured against independent ground truth, on images the model never trained on. v1.2 trains on 74 of the 87 Manga109-s titles, so Japanese is measured two ways: on JMangaBench_Read, recent doujin pages from a different source, and on the 13 Manga109-s titles held out from training (listed in manga109s_split.json). For Chinese and English we built a fresh held-out set from galleries published after the training cutoff (IDs above the training maximum — zero overlap) and re-annotated every crop by hand, with the teacher output hidden. lCER is the lenient character-error-rate (NFKC, whitespace stripped, decorative symbols dropped, dot-runs unified), lower is better.

Japanese, JMangaBench_Read v1.3.1 (7,789 boxes + 380 sound effects)

Every row is scored by the benchmark's own scorer (exact-match %, by box type): the others from the predictions the benchmark publishes, Baberu v1.2 run here with the published decode. † is the benchmark authors' own model; they used an earlier version of the benchmark while developing it (their conflict-of-interest note says so).

On recent manga it never saw, v1.2 is level with the 0.9B manga-tuned VLM overall (it would sit 10th on the leaderboard, where v1.1 is 13th) and ahead of it on handwriting and sound effects, the two box types the Manga109-s human labels added to training. Printed dialogue barely moved, because that is what the teacher already read well. Hayai v2.5 stays ahead of Baberu on every column.

Japanese, Manga109-v2026, the 13 titles held out from training

Dialogue n=3,000 (the v2026 annotations), sound effects n=2,000 (the COO annotations, cropped to each polygon's box). Both lCER columns are capped at 1 per box: PaddleOCR-VL looped once, writing イ 3,722 times, and that one box alone would have lifted its mean to 0.148. It was not run on the sound effects. manga-ocr was trained on Manga109-s, so these titles are likely in its training data; neither Baberu saw them. Sound effects are where manga-ocr's labels could not help: it was never trained on them either.

The 2,000-crop Manga109 set that earlier versions of this card used samples all 87 titles, so it no longer measures v1.2 fairly; v1.1's numbers on it stay on its card (--revision v1.1).

Chinese and English, hand-annotated held-out

Two ways to count, because they disagree and the disagreement is the result. Per-bubble averages the error over each bubble (the unit a bubble reader delivers); char-weighted weights by total characters (long passages dominate). PaddleOCR-VL is Baberu's Chinese/English teacher.

Per bubble, the 115M student wins both languages and gets more bubbles exactly right; it also never melted into a repetition loop, while the 0.9B did on two short English bubbles. Per character, the 0.9B still wins Chinese, where it reads long passages better. Call it toe-to-toe with a model 8× its size: ahead and more reliable on the typical bubble, behind on long Chinese text. Against v1.1 (zh 0.060 / en 0.054 per bubble), neither change is significant: the 95% paired-bootstrap intervals include zero. The English bubbles that got worse are short ones read as Japanese (AH♥ → み), the price of a stream that is now more Japanese.

Long bubbles (v1.1)

Up to v1.0, every training label was cut at 64 characters and the cut label still ended in end-of-text. About 8% of English labels are longer than that (Japanese and Chinese bubbles rarely are), so the model learned to stop right after character 64. v1.1 trains the decoder further on labels up to 256 characters, with no end-of-text after a cut and without the teacher's own repetition loops, which made up a quarter of the long Japanese labels and half of the Chinese ones. The vision encoder and projector stayed frozen, so the vision ONNX files are byte-identical.

English bubbles of 65–256 characters, from two shards the v1.1 run never read, scored against the teacher's labels with the shipped ONNX build (vision_fp16 + decoder_*_int8):

Past about 190 characters the end of the bubble still comes out garbled. At 224×224 the letters get too small to read, which is a limit of the input size rather than of training. v1.2 keeps the long-bubble reading: on the same held-out English bubbles over 64 characters, lCER 0.026 against v1.1's 0.027.

On short bubbles the numbers dipped slightly, within noise: against v1.0 on the independent sets this card used then, Manga109 lCER moved 0.0344 → 0.0359 and Chinese per-bubble lCER 0.052 → 0.060, and the 95% paired-bootstrap interval of each difference includes zero. Reading further also needed one more guard in the published decode: a run of one symbol (ー, ～, ・) now stops at 16, as one letter or kana already stopped at 12. Without it, a stretched ー on one Manga109 page ran to the token limit.

Where it loses

Public, never-seen gold labels from COMICS Text+ (1940s Western comics, n=480):

On vintage Western comics it has never seen, the general 0.9B model reads cleaner. Baberu is tuned for modern manga, and this gap marks the edge of that focus rather than hiding it. It is also the one set where v1.2 is measurably worse than v1.1 (the 95% paired-bootstrap interval of the difference is +0.003 to +0.021): the Japanese-heavy v1.2 run reads 8 of these 480 English boxes as Japanese, against 4 for v1.1. On modern English bubbles that count did not change (3 of 223 for both), so for old Western lettering --revision v1.1 is the better choice. (Earlier versions of this card listed 0.149 here, measured on v1.0 with its 64-character limit.) Closer to home, Hayai v2.5 still reads recent handwriting better (76.6% against 67.3% exactly right on JMangaBench_Read), and the 0.9B VLM reads printed text more exactly.

Quick start

The input is one speech-bubble crop (run a text detector upstream), not a full page.

ONNX, the small path (no PyTorch):

pip install onnxruntime numpy pillow
huggingface-cli download genshiai-daichi/baberu-ocr --local-dir baberu-ocr
cd baberu-ocr

python onnx_infer.py --image bubble.jpg                          # 243 MB, lossless
python onnx_infer.py --vision vision_int4.onnx --image bubble.jpg # 122 MB, smallest

onnx_infer.py carries its own preprocessing, KV-cache decode loop, and the published decode settings (greedy, repetition_penalty=1.2, and run caps of 12 for one letter or kana and 16 for one symbol), so its output matches the PyTorch model.

PyTorch, for fine-tuning or transformers:

pip install torch transformers safetensors pillow
huggingface-cli download genshiai-daichi/baberu-ocr --local-dir baberu-ocr
cd baberu-ocr
python inference.py --model . --image bubble.jpg

To fine-tune the released weights on your own bubbles, see TRAINING.md: train_ocr.py --finetune-from . continues training with a fresh optimizer (add --unfreeze-vision to adapt the encoder too). The full from-scratch recipe (train_text.py → train_ocr.py) is included as well.

Sizes and quantization (ONNX)

Two tiers are recommended, both with the decoder at int8:

Japanese columns are the Manga109-s held-out titles and JMangaBench_Read from Benchmarks. The measured sweep (quantization-results.md) found two things worth knowing. Decoder int8 is close to free here, because greedy decoding shrugs off weight-quantization noise. The cost lives in vision, and even there int4 weight-only beats int8-dynamic on both size and accuracy. The dominated variants (vision int8-dynamic, decoder int4) are not shipped.

Limitations

• One bubble crop in, text out. You supply the text-detection stage. This is not page-level OCR.
• It confuses dakuten and handakuten (ポ vs ボ), misses some rare kanji, and drops the occasional small kana.
• Handwriting on recent manga is still its weakest Japanese box type (67% exactly right on JMangaBench_Read). The human labels it learned sound effects and handwriting from come from Manga109, drawn between the 1970s and the 2010s.
• A short English bubble next to Japanese text is sometimes read as Japanese.
• A bubble of more than about 190 characters reads well at the start and garbles toward the end (the 224×224 input is too coarse for that much text).
• On print far outside its domain, a general large VLM reads better.

Training data and license

Trained on an independently collected corpus of manga speech-bubble crops, with text labels distilled from manga-ocr (Japanese) and PaddleOCR-VL (Chinese, English), plus, from v1.2, human labels from Manga109-s.

This model was trained using the Manga109-s dataset: 74 of its 87 titles, dialogue and narration from the Manga109-v2026 annotations and sound effects from the COO annotations. The other 13 titles are held out for evaluation (manga109s_split.json; build_manga109s_split.py cuts the crops from your own copy). Manga109-s permits machine-learning use and commercial use of the results; its conditions include stating that use, as here, and not redistributing the dataset, so no Manga109 image ships here.

The training images are not redistributed; only the weights and code ship here. License is Apache-2.0 for the weights and code, matching the upstream components (DINOv2, manga-ocr, and PaddleOCR are all Apache-2.0).

Acknowledgements

• DINOv2 (Meta), the vision encoder
• manga-ocr, the de-facto Japanese manga OCR baseline and the Japanese teacher
• PaddleOCR-VL, the Chinese and English teacher
• Manga109 (Aizawa Yamasaki Matsui Lab, The University of Tokyo) and COO, the human labels behind v1.2
• JMangaBench_Read, the independent Japanese benchmark

Citation

Manga109-s, used for v1.2's training labels:

@inproceedings{baek2026mangav26,
  title     = {{Manga109-v2026: Revisiting Manga109 Annotations for Modern Manga Understanding}},
  author    = {Baek, Jeonghun and Miyai, Atsuyuki and Onohara, Shota and Ikuta, Hikaru and Aizawa, Kiyoharu},
  booktitle = {Culture × AI Workshop at ICML 2026},
  year      = {2026},
}
@article{multimedia_aizawa_2020,
  author  = {Kiyoharu Aizawa and Azuma Fujimoto and Atsushi Otsubo and Toru Ogawa and Yusuke Matsui and Koki Tsubota and Hikaru Ikuta},
  title   = {Building a Manga Dataset ``Manga109'' with Annotations for Multimedia Applications},
  journal = {IEEE MultiMedia}, volume = {27}, number = {2}, pages = {8--18}, year = {2020},
  doi     = {10.1109/mmul.2020.2987895},
}
@article{mtap_matsui_2017,
  author  = {Yusuke Matsui and Kota Ito and Yuji Aramaki and Azuma Fujimoto and Toru Ogawa and Toshihiko Yamasaki and Kiyoharu Aizawa},
  title   = {Sketch-based Manga Retrieval using Manga109 Dataset},
  journal = {Multimedia Tools and Applications}, volume = {76}, number = {20}, pages = {21811--21838}, year = {2017},
  doi     = {10.1007/s11042-016-4020-z},
}
@inproceedings{baek2022COO,
  title     = {COO: Comic Onomatopoeia Dataset for Recognizing Arbitrary or Truncated Texts},
  author    = {Baek, Jeonghun and Matsui, Yusuke and Aizawa, Kiyoharu},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2022},
}