Model Card for vit_reg4_so150m_p14_ls_dino-v2-bio

vit_reg4_so150m_p14_ls_dino-v2-bio is a Bio-DINO image encoder for natural photographs of living organisms. It uses a SoViT-150M/14 Vision Transformer with 4 register tokens and 133.6M backbone parameters, trained with a DINOv2-style self-supervised objective on approximately 31 million curated images spanning plants, fungi, insects, fish, corals, birds, mammals and other biodiversity subjects.

The model is released as a single repository with three checkpoints: 224 x 224, 252 x 252 and 336 x 336. The 252px checkpoint is the recommended default, the 224px checkpoint is a faster and lower-cost variant, and the 336px checkpoint is a short high-resolution adaptation for users who want to trade latency for more input detail. All checkpoints are intended primarily as frozen image encoders for embeddings, retrieval, clustering, probing and transfer learning, rather than as ready-made species classifiers.

This repository releases the Birder backbone checkpoints for direct use as image encoders. The full DINO training weights, including the DINO head and training state needed for continued self-supervised work, are released separately in the full DINO training weights companion repository.

Model Details

• Model Type: Image encoder and detection backbone

• Model Stats:

  • Params (M): 133.6
  • Input image size: 252 x 252
  • Additional released sizes: 224 x 224, 336 x 336

• Dataset: Trained on a diverse dataset of approximately 31M images, including:

  • TreeOfLife-10M-EOL-NaturalImages
  • iNaturalist 2021
  • BIOSCAN-5M (pretrain split)
  • TreeOfLife-200M (subset)
  • IP102 v1.1
  • iWildCam 2022 (subset)
  • The Birder dataset (private dataset)

• Papers:

  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: https://arxiv.org/abs/2010.11929
  • Vision Transformers Need Registers: https://arxiv.org/abs/2309.16588
  • Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design: https://arxiv.org/abs/2305.13035
  • DINOv2: Learning Robust Visual Features without Supervision: https://arxiv.org/abs/2304.07193

Released Checkpoints / Model Variants

All three backbone checkpoints are released in this model repository.

Intended Use And Scope

This model is intended for natural photographs of living organisms and biodiversity subjects: organismal field photography, camera-trap imagery, collection-style photographs, underwater photography and similar real-world images of living systems. It is best used as a general-purpose image encoder for representation-learning workflows, especially when the downstream task benefits from visual similarity, fine-grained biological structure or transfer from broad biodiversity imagery. Image retrieval and visual similarity search are primary use cases for Bio-DINO. The model was evaluated heavily as a frozen embedding model and is particularly well suited to finding visually related biological images across large collections.

Suitable Uses

Suitable uses include:

• Image embedding extraction for natural photographs of living organisms.
• Nearest-neighbor retrieval, visual similarity search and reference-image matching.
• Clustering, dataset exploration, and biodiversity image organization.
• Dataset deduplication, near-duplicate review, and collection triage.
• Frozen-embedding evaluation with linear probes, k-NN, SimpleShot, SVM, MLP probes, or retrieval metrics.
• Lightweight supervised transfer for taxonomic, ecological, trait, or visual-recognition tasks.
• Fine-tuning or initialization for downstream biodiversity image models.
• Continued self-supervised training and representation-learning research.

Out-of-Scope Domains

The model is not intended for microscopy, pathology, medical imaging, X-ray/radiology, satellite imagery, maps, diagrams, charts, drawings, document understanding or other non-photographic visual domains. It was trained for natural photography, not controlled lab imagery, scientific instrument imagery or clinical imagery.

Training Data

The model was trained on approximately 31 million curated images focused on natural photographs of living organisms and biodiversity subjects. The training mixture combines public biodiversity datasets, selected subsets of larger public image collections, and curated private Birder data sources. The goal of the corpus was broad visual coverage of the biological world rather than a fixed taxonomy, a supervised class list or a balanced benchmark-style dataset.

At a high level, the curation process follows the same concept as the Birder Vision Data Curation walkthrough for TreeOfLife-10M: input validation, near-duplicate removal, example-based filtering of non-natural images, quality/aesthetic filtering, and diversity-oriented sampling. The filtering process was designed to emphasize real photographs of living organisms and reduce illustrations, documents, maps, charts, diagrams, screenshots, lab imagery, medical imagery and other content outside the intended domain.

The final corpus includes broad biological coverage across plants, fungi, insects, fish, corals, birds, mammals and other organismal groups.

Known public components and source families used in the training mixture include:

• TreeOfLife-10M-EOL-NaturalImages
• iNaturalist 2021
• BIOSCAN-5M pretrain split
• TreeOfLife-200M subset
• IP102 v1.1
• iWildCam 2022 subset

Training Procedure

The model was trained with a DINOv2-style self-supervised objective using a SoViT-150M/14 backbone with 4 register tokens. The longest training stage was run at 224 x 224 for efficiency, then continued at 252 x 252 for the main release checkpoint and finished with a short 336 x 336 high-resolution adaptation.

Resolution Schedule

Detailed launch/configuration files for the self-supervised runs are published with the release. The 224px launch/configuration file is available here: 224px Slurm file.

Evaluation Overview

The evaluation is designed to assess representation quality across diverse biological and biodiversity image tasks.

Evaluation protocols vary by dataset. They are intended to test whether frozen representations are useful for separability, neighborhood structure, trait prediction, retrieval, clustering and lightweight transfer, rather than to optimize an end-to-end classifier for a single benchmark. For this reason, the benchmarks use simple probes and non-parametric methods over frozen features wherever possible.

The reference set includes biology-focused CLIP models such as BioCLIP v1, BioCLIP v2 and BioCLIP v2.5, generic CLIP/VLM models such as PE-Core and SigLIP, generic self-supervised encoders such as DINOv2 and DINOv3, a supervised ConvNeXt ImageNet-22K reference and the spatially focused TIPS-v2 model. These references are included to contextualize embedding quality across different training paradigms, not as a controlled ablation of architecture, data or supervision.

Evaluation Protocols

Benchmark Results

The benchmark suite was used both to select release checkpoints and to understand how the representation improved during training. The charts below show Bio-DINO checkpoints across training epochs against contextual reference models. The three released checkpoints correspond to 224px at epoch 185, 252px at epoch 245, and 336px at epoch 250.

Aggregate benchmark progression over Bio-DINO training. Points show Bio-DINO checkpoints across training epochs, with the released 224px, 252px and 336px checkpoints marked by stars. Dashed lines show reference encoders evaluated under the same local protocols. The aggregate score is computed by normalizing each benchmark against the reference-model range and averaging across benchmarks, so it should be read as a relative summary of this evaluation suite rather than an absolute measure of biological expertise.

CUB-200-2011 retrieval benchmark progression over Bio-DINO training. Points show Bio-DINO checkpoints across training epochs, with the released 224px, 252px, and 336px checkpoints marked by stars. The metric is mean average precision, using frozen image embeddings for visual retrieval.

NeWT benchmark progression. Accuracy is measured with SVM classifiers over the NeWT binary task suite using frozen image embeddings.

SnakeCLEF2023 benchmark progression. Accuracy is measured with a linear probe over frozen image embeddings.

The iNaturalist 2021 result uses a fuller linear-probing setup than the frozen-embedding evaluations above: the backbone is frozen, but the classification head is trained for multiple epochs with light image augmentation.

For the iNaturalist 2021 linear-probe table, parameter counts include the frozen image encoder plus the trained 10k-class linear classification head.

The reference models are included for context only. They differ in architecture, model size, pretraining data, supervision type, intended use, preprocessing and input resolution, so these comparisons should be read as local benchmark context rather than as controlled ablations.

Results reflect the local protocols above, not absolute biological expertise. They may change with preprocessing, crop policy, input resolution, probe hyperparameters, random seed and embedding extraction layer.

Efficiency / Latency vs Quality

The three Bio-DINO checkpoints expose a practical speed/quality tradeoff. The 252px checkpoint is the recommended default, the 224px checkpoint is faster and lower-cost, and the 336px checkpoint can improve some detail-sensitive tasks at a substantial latency cost.

The figure below shows the CUDA latency vs accuracy Pareto view from the iNaturalist 2021 linear-probe evaluation. It is intended as a deployment-oriented view of the same kind of tradeoff shown in the broader evaluation above.

Accuracy/latency tradeoff on the iNaturalist 2021 linear-probe evaluation. Accuracy is measured with a frozen image encoder and trained classification head. Latency is measured as CUDA milliseconds per sample with torch.compile, batch size 512 and AMP enabled. Bio-DINO checkpoints are shown at 224px, 252px and 336px, illustrating the tradeoff between input resolution, accuracy and inference cost.

The full interactive latency/quality chart is available in the Birder Leaderboard Space.

Relationship To BioCLIP And Other Work

BioCLIP, BioCLIP 2 and BioCLIP 2.5 are important reference points for this release. They advanced biological image understanding with large-scale biodiversity-focused CLIP training, strong Tree of Life image-text datasets and open model releases. They remain very strong baselines in the evaluations reported here, especially on fine-grained biological recognition tasks.

Bio-DINO is intended to be complementary. It is an image-only, self-supervised encoder released for embedding extraction, image retrieval, transfer learning and continued DINO-style research. It does not include a text encoder and should not be read as a replacement for BioCLIP-style image-text models. The companion release of the full DINO training weights is intended to make the training state reusable for researchers who want to continue or adapt the self-supervised training process. The full training state is available in the full DINO training weights companion repository.

Model Usage

Image Embeddings

import birder
from birder.inference.classification import infer_image

# Option 1: manual setup (more control over preprocessing)
net, model_info = birder.load_pretrained_model("vit_reg4_so150m_p14_ls_dino-v2-bio-252px", inference=True)

# Get the image size the model was trained on
size = birder.get_size_from_signature(model_info.signature)

# Create an inference transform
transform = birder.classification_transform(size, model_info.rgb_stats)

# Option 2: helper (quick start with default preprocessing)
net, model_info, transform = birder.load_pretrained_model_and_transform("vit_reg4_so150m_p14_ls_dino-v2-bio-252px", inference=True)

image = "path/to/image.jpeg"  # or a PIL image
out, embedding = infer_image(net, image, transform, return_embedding=True)
# embedding is a NumPy array with shape of (1, 896)

Detection Feature Map

from PIL import Image
import birder

net, model_info, transform = birder.load_pretrained_model_and_transform("vit_reg4_so150m_p14_ls_dino-v2-bio-252px", inference=True)

image = Image.open("path/to/image.jpeg")
features = net.detection_features(transform(image).unsqueeze(0))
# features is a dict (stage name -> torch.Tensor)
print([(k, v.size()) for k, v in features.items()])
# Output example:
# [('stage1', torch.Size([1, 896, 18, 18]))]

Citation

@misc{dosovitskiy2021imageworth16x16words,
      title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
      author={Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby},
      year={2021},
      eprint={2010.11929},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2010.11929},
}

@misc{darcet2024visiontransformersneedregisters,
      title={Vision Transformers Need Registers},
      author={Timothée Darcet and Maxime Oquab and Julien Mairal and Piotr Bojanowski},
      year={2024},
      eprint={2309.16588},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2309.16588},
}

@misc{alabdulmohsin2024gettingvitshapescaling,
      title={Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design},
      author={Ibrahim Alabdulmohsin and Xiaohua Zhai and Alexander Kolesnikov and Lucas Beyer},
      year={2024},
      eprint={2305.13035},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2305.13035},
}

@misc{oquab2024dinov2learningrobustvisual,
      title={DINOv2: Learning Robust Visual Features without Supervision},
      author={Maxime Oquab and Timothée Darcet and Théo Moutakanni and Huy Vo and Marc Szafraniec and Vasil Khalidov and Pierre Fernandez and Daniel Haziza and Francisco Massa and Alaaeldin El-Nouby and Mahmoud Assran and Nicolas Ballas and Wojciech Galuba and Russell Howes and Po-Yao Huang and Shang-Wen Li and Ishan Misra and Michael Rabbat and Vasu Sharma and Gabriel Synnaeve and Hu Xu and Hervé Jegou and Julien Mairal and Patrick Labatut and Armand Joulin and Piotr Bojanowski},
      year={2024},
      eprint={2304.07193},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2304.07193},
}

Acknowledgements

This release builds on the broader open-source vision and biodiversity machine-learning ecosystem. In particular, the BioCLIP, BioCLIP 2 and BioCLIP 2.5 work by the Imageomics team helped move biological image foundation models forward and provided strong open reference models for this release.

This work benefited from computational resources provided by The Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, through a collaboration with the STARdbi project led by Chen Keasar. The released models and evaluations are intended to support biodiversity computer-vision research, including visual identification, classification, retrieval, and diversity analysis of insects and other organisms.