PhenoVision: Automated Plant Reproductive Phenology from Field Images

PhenoVision is a Vision Transformer (ViT-Large) model fine-tuned to detect flowers and fruits in plant photographs. It was trained on 1.5 million human-annotated iNaturalist images and has been used to generate over 30 million new phenology records across 119,000+ plant species, vastly expanding global coverage of plant reproductive phenology data.

Model Details

• Model type: Multi-label image classification (sigmoid outputs)
• Architecture: Vision Transformer Large (ViT-L/16), ~304M parameters
• Input: 224 x 224 RGB images
• Output: 2 logits (flower, fruit) — apply sigmoid for probabilities
• Pretraining: PlantCLEF 2022 checkpoint ("virtual taxonomist" — trained on 2.9M plant species images)
• Current version: v1.1.0
• Model DOI: 10.57967/hf/7952
• Developer: Phenobase
• Repository: github.com/Phenobase/phenovision
• License: MIT

Key Innovation: Virtual Taxonomist Pretraining

Instead of standard ImageNet pretraining, PhenoVision uses a ViT-Large checkpoint pretrained on the PlantCLEF 2022 dataset (2.9 million plant images for species classification). Since species classification relies heavily on recognizing reproductive structures (flowers, fruits), this domain-specific pretraining provides a strong initialization for phenology detection. Compared to ImageNet pretraining, PlantCLEF pretraining achieved:

• Higher accuracy: TSS = 0.864 vs. 0.835
• Faster convergence: Best epoch at 4 vs. 11

Intended Uses

Primary use: Detecting the presence of flowers and/or fruits in field photographs of plants.

Suitable for:

• Automated phenology annotation of iNaturalist and other community science images
• Large-scale phenology monitoring and climate change research
• Generating presence-only reproductive phenology datasets
• Integration with phenology databases (e.g., Phenobase, USA-NPN)

Out of scope:

• Counting individual flowers or fruits
• Distinguishing flower developmental stages (buds vs. open vs. senescent)
• Detecting leaf phenology (use PhenoVisionL instead)
• Identifying plant species (this is a phenology model, not a taxonomic classifier)

How to Use

Requires transformers, torch, and torchvision (torchvision is needed by the image processor in transformers v5+).

from transformers import AutoModelForImageClassification, AutoImageProcessor
from PIL import Image
import torch

# Load model and processor
processor = AutoImageProcessor.from_pretrained("phenobase/phenovision")
model = AutoModelForImageClassification.from_pretrained("phenobase/phenovision")
model.eval()

# Run inference
image = Image.open("plant_photo.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.sigmoid(outputs.logits)[0]

# IMPORTANT: output order is [fruit, flower] — index 0 is FRUIT, index 1 is FLOWER.
fruit_prob  = probs[0].item()
flower_prob = probs[1].item()

print(f"Flower: {flower_prob:.3f}")
print(f"Fruit:  {fruit_prob:.3f}")

Applying Thresholds

Raw probabilities should be converted to detection calls using the optimized thresholds and uncertainty buffers provided as companion files. Predictions falling within the buffer zone are classified as "Equivocal" and should be excluded for research-quality outputs.

• Probability above (threshold + buffer_upper) → Detected (high certainty)
• Probability below (threshold - buffer_lower) → Not Detected (high certainty)
• Probability within buffer zone → Equivocal (exclude from analysis)

Training Data

• Source: iNaturalist open data (research-grade observations)
• Size: 1,535,930 images from 119,340 species across 10,406 genera and 408 plant families
• Splits: 60% train (921,720) / 20% validation (307,291) / 20% test (306,919), stratified by genus
• Annotations: Human phenology annotations from iNaturalist platform (reproductiveCondition field)
• Licensing: Images under CC-0, CC-BY, or CC-BY-NC licenses
• Note: Approximately 1-5% of training annotations are marked "unknown" due to annotation difficulty

Training Procedure

• Optimizer: AdamW
• Learning rate: 5e-4 (base), with layer-wise decay factor 0.65
• Batch size: 384
• Weight decay: 0.05
• Data augmentation: RandAugment
• Epochs: 10 (best model selected at epoch 7 by average Data Quality Index)
• Hardware: NVIDIA A100 GPU
• Loss: Binary cross-entropy (multi-label)
• v1.1.0 training: Fine-tuned from v1.0.0 checkpoint on updated data snapshot (2025-10-27)

Evaluation Results

Test Set Performance (v1.1.0)

Expert Validation

Independent expert review of model predictions:

• Flower presence: 98.6% agreement
• Fruit presence: 90.4% agreement

Taxonomic Coverage

• Species: 119,340 from 10,406 genera and 408 families
• Genera with 10+ records: 7,409 (flowers), 5,240 (fruits)
• Median records per genus: 184 (flowers), 85 (fruits)
• New geographic grid cells: 3,798 (flowers), 4,147 (fruits) with no prior phenology data

Companion Files

The following files are uploaded alongside the model weights:

Limitations and Biases

Design Limitations

• Presence-only: The model reports detections but NOT absences. A non-detection does not mean the plant lacks flowers/fruits — it may simply not be visible in the image.
• Partial plant coverage: Images typically show only part of a plant. Reproductive structures may exist on non-photographed parts.
• Buffer zone data loss: Applying uncertainty thresholds removes ~7-9% of predictions as equivocal, trading completeness for accuracy.

Known Failure Modes

• Inconspicuous reproductive structures (grasses, sedges) are harder to detect
• Flower buds may be confused with open flowers
• Background plants with flowers/fruits can cause false positives for the focal plant
• Some families show lower accuracy (e.g., Haloragaceae ~79%)

Data Biases

• Reflects iNaturalist's geographic biases: overrepresentation of urban areas, developed countries, and coastal regions
• Taxonomic bias toward common, conspicuous species
• Limited coverage in biodiversity-rich tropical regions

Annotation Quality

• Training labels come from community science annotations with inherent variability
• Some iNaturalist annotations are incomplete (e.g., flower present but only fruit annotated)
• Family-level accuracy statistics (in family_stats.csv) should be consulted when interpreting results for specific taxonomic groups

Citation

If you use PhenoVision in your research, please cite:

@article{dinnage2025phenovision,
  title={PhenoVision: A framework for automating and delivering research-ready plant phenology data from field images},
  author={Dinnage, Russell and Grady, Erin and Neal, Nevyn and Deck, Jonn and Denny, Ellen and Walls, Ramona and Seltzer, Carrie and Guralnick, Robert and Li, Daijiang},
  journal={Methods in Ecology and Evolution},
  volume={16},
  pages={1763--1780},
  year={2025},
  doi={10.1111/2041-210X.14346}
}

Acknowledgments

• Funding: National Science Foundation (NSF)
• Data: iNaturalist community and platform
• Infrastructure: Phenobase — a global plant phenology database
• Integration: Plant Phenology Ontology (PPO), USA National Phenology Network (USA-NPN)