fisheye8k_microsoft_conditional-detr-resnet-50

This model is a fine-tuned version of microsoft/conditional-detr-resnet-50 on the Voxel51/fisheye8k dataset. It is a key artifact of the Mcity Data Engine project.

• Paper: Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection
• Project Page: Mcity Data Engine Docs
• Code: GitHub Repository for Mcity Data Engine

It achieves the following results on the evaluation set:

• Loss: 1.4466

Model description

This model is an object detection model fine-tuned on the microsoft/conditional-detr-resnet-50 architecture using the Fisheye8K dataset. It is developed as part of the Mcity Data Engine, an open-source framework designed to facilitate iterative model improvement through open-vocabulary data selection. The Mcity Data Engine addresses the challenge of selecting and labeling appropriate samples for machine learning models, particularly for detecting long-tail and rare classes of interest in large amounts of unlabeled data within Intelligent Transportation Systems (ITS). This fine-tuned model demonstrates the practical application of the Data Engine's capabilities in enhancing roadside perception systems for autonomous driving and smart city applications.

The model is trained to detect specific categories, as defined in its configuration: Bus, Bike, Car, Pedestrian, and Truck.

Intended uses & limitations

This model is intended for object detection tasks within Intelligent Transportation Systems, specifically for identifying vehicles and vulnerable road users in visual data, such as that collected from fisheye cameras. Its primary use case is within the Mcity Data Engine framework for research and development related to improving perception models with rare and novel data.

Limitations:

• The model's performance may vary on data significantly different from the Fisheye8K dataset (e.g., different camera types, environments, or lighting conditions).
• Like all deep learning models, it may exhibit biases present in the training data and may not generalize perfectly to all real-world scenarios.
• Further evaluation on diverse real-world ITS data is recommended for specific deployment scenarios.

Usage

You can use this model directly with the Hugging Face transformers library for object detection tasks.

from transformers import AutoImageProcessor, AutoModelForObjectDetection
import torch
from PIL import Image
import requests

# Load image processor and model
model_name = "mcity-data-engine/fisheye8k_microsoft_conditional-detr-resnet-50"
processor = AutoImageProcessor.from_pretrained(model_name)
model = AutoModelForObjectDetection.from_pretrained(model_name)

# Example image (replace with your own image URL or local path)
url = "http://images.cocodataset.org/val2017/000000039769.jpg" # A standard COCO image
image = Image.open(requests.get(url, stream=True).raw).convert("RGB") # Ensure RGB format

# Perform inference
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)

# Post-process and print results
target_sizes = torch.tensor([image.size[::-1]]) # (height, width)
results = processor.post_process_object_detection(outputs, target_sizes=target_sizes, threshold=0.9)[0] # Apply a confidence threshold

print("Detected objects:")
for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
    box = [round(i, 2) for i in box.tolist()]
    print(
        f"  - {model.config.id2label[label.item()]} with confidence "
        f"{round(score.item(), 3)} at location {box}"
    )

Training and evaluation data

This model was fine-tuned on the Voxel51/fisheye8k dataset. The Fisheye8K dataset contains images captured from fisheye cameras, primarily focusing on intelligent transportation system scenarios.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 0
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• num_epochs: 36
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.48.3
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0

Acknowledgements

Mcity would like to thank Amazon Web Services (AWS) for their pivotal role in providing the cloud infrastructure on which the Data Engine depends. We couldn’t have done it without their tremendous support!

Citation

If you use the Mcity Data Engine in your research, feel free to cite the project:

@article{bogdoll2025mcitydataengine,
  title={Mcity Data Engine},
  author={Bogdoll, Daniel and Anata, Rajanikant Patnaik and Stevens, Gregory},
  journal={GitHub. Note: https://github.com/mcity/mcity_data_engine},
  year={2025}
}