Weed and crop instance segmentation

Four-class Mask R-CNN with a Swin-S backbone. This version contains the best validation-mask checkpoint (epoch 33) from the 42-epoch run with adaptive crops, shuffled training order, and class-balanced image repetition.

Measured test performance

Evaluation used 186 test images/tiles from 126 held-out original photos, COCO IoUs 0.50:0.05:0.95, score threshold 0.001, and maxDets [1, 10, 100]. Original photos were split before cropping. These are crop-level results; whole-photo inference in the demo uses a different input scale. Full results are in evaluation/results.json.

Usage

Use a compatible MMDetection 3.x environment with MMCV 2.1.0, MMEngine 0.10.x, PyTorch, torchvision, transformers, safetensors, Pillow, and NumPy. Validation of this export used PyTorch 2.4.1 and transformers 4.46.3. The existing BAT102 Docker environment uses Python 3.8 and MMDetection 3.x.

import torch
from PIL import Image, ImageOps
from transformers import AutoImageProcessor, AutoModelForObjectDetection

repo = "haiquanli/weed_swin"
model = AutoModelForObjectDetection.from_pretrained(repo, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(repo, trust_remote_code=True)
with Image.open("plants.jpg") as source:
    image = ImageOps.exif_transpose(source).convert("RGB")
inputs = processor(images=image, return_tensors="pt")
inputs["pixel_values"] = inputs["pixel_values"].to(model.device)
with torch.inference_mode():
    outputs = model(**inputs, score_threshold=0.25)
predictions = processor.post_process_object_detection(outputs, threshold=0.25)[0]
# predictions: boxes, masks, scores, labels in the original image coordinates

The processor preserves aspect ratio and supplies raw BGR pixels; MMDetection performs its trained color conversion, normalization, and divisor padding. It returns full-resolution instance masks. Weights are loaded strictly against the exported architecture, and the bridge pins all model files to one Hub commit.

For benchmark evaluation use score_threshold=0.001 and export the results using the training COCO category mapping [1, 2, 4, 5]. Model labels themselves are contiguous indices [0, 1, 2, 3] as shown above.

Adaptive crop profiles are in crop_profiles.json. A profile's mean object size is fitted only on training photos as mean sqrt(box width × box height). Choose a source profile for new images without using their ground-truth labels. The demo accepts whole images; it does not automatically choose a source profile.

The previous fixed-crop and adaptive-only models scored 52.9% and 52.1% mask mAP, respectively. The current combined training policy scored 56.1%; shuffling, sampling, loss weights and optimizer-update counts changed together, so their individual contributions have not been isolated.