OCR Captcha Router

Source code, install instructions, and full documentation: github.com/khabbazan/ocr-captcha-router

A lightweight captcha router that classifies an image as digits or math and sends it to the matching TrOCR expert. Experts are downloaded on first use and cached by Hugging Face. Uncertain routes fall back to running both experts and choosing the candidate that satisfies its grammar.

Model details

• Architecture: ConvNeXt-tiny image classifier (2 labels: digits, math)
• Base model: facebook/convnext-tiny-224
• Router input: aspect-preserving letterbox into 96x384 (parameters in router_preprocess.json)
• Experts: two TrOCR models, one for digit captchas and one for math captchas

Source models

How it works

1. Route: the router predicts digits or math from the letterboxed image.
2. Run the expert: the selected TrOCR model transcribes the image. Digit output is validated against ^\d{4,7}$ over up to three preprocessed views (original, autocontrast grayscale, inverted). Math output is repaired from a truncated a+b= to a+b=?.
3. Fallback: if router confidence is below 0.95 or the primary output fails its grammar, the other expert also runs and the valid candidate is chosen. If neither is valid the result is route: "unknown" and valid_format: false.

Evaluation

The router was trained on synthetic captchas of both classes, rendered in shared styles so that the text content, not the style, separates the classes. It was validated on live captchas.

These scores measure routing only, not the text read by the experts. The real validation set was used to choose the checkpoint, so its score is optimistic.

Usage

from resolver import CaptchaResolver

resolver = CaptchaResolver.from_pretrained("arkhabbazan/ocr-captcha-router")
print(resolver.predict("captcha.jpeg"))

predict accepts a path or a PIL.Image and returns the text, the route, the router confidence and probabilities, whether the dual-model fallback ran, and whether the output matched its grammar. To use only the router:

from router_inference import CaptchaRouter

router = CaptchaRouter("path/to/downloaded/repo")
label, confidence, probabilities = router.route(image)

Keep the captcha at its original aspect ratio. The resolver letterboxes only the router input and leaves OCR preprocessing to each TrOCR processor. The first request for each route downloads that expert; later requests reuse the in-memory model and the Hugging Face cache. math_confidence_threshold (0.0 to 1.0) is the minimum router probability required to choose the math result when both expert outputs match their valid formats.

Limitations

• The router and both experts were trained mostly on synthetic data.
• Unseen fonts and layouts may reduce routing and OCR accuracy.
• Routing accuracy does not guarantee that the digits or math text is read correctly.
• Usage must be authorized and compliant with applicable policies.