VisualHeist - figure, scheme and table segmentation from PDFs (with captions, headers & footnotes)

Model Summary

VisualHeist is an object detection model finetuned to extract tables and figures from PDFs. VisualHeist has two versions:

• visualheist-base[HF] (0.23B)
• visualheist-large[HF] (0.77B)

The base model is recommended if you are running it on low-RAM systems

The models are finetuned from microsoft/Florence-2 checkpoints. VisualHeist is inspired by and adapted from yifeihu/TF-ID

• The models were finetuned with 3435 figures and 1716 tables from 110 PDF articles across various publishers. All bounding boxes are manually annotated using CoCo Annotator.
• TF-ID models take an image of a single paper page as the input, and return image files for all figures, schemes and tables in the given page.

Training Code and Dataset

• Dataset: Zenodo repository
• Code: github.com/aspuru-guzik-group/MERMaid

Benchmarks

We manually curated a diverse evaluation dataset consisting of 121 literature articles covering a range of topics, including organic and inorganic chemistry, atmospheric science, batteries, materials science, metal-organic frameworks (MOFs), biology, and science education. These PDFs, published between 1949 and 2025, include both main articles and supplementary materials.

We also additionally curated another collection of 98 literature articles (MERMaid-100) reporting novel reaction methodologies that spans three distinct chemical domains: organic electrosynthesis, photocatalysis, and organic synthesis.

Additional performance discussion can be found from our preprint article

The full DOI lists can be downloaded from ourZenodo repository.

The evaluation results for visualheist-large are:

Running the Model

Refer to our github repository for detailed instructions on how to run the model

BibTex and citation info

<To be updated with our archive citation>