NoctOWL: Fine-Grained Open-Vocabulary Object Detector

Model Description

NoctOWL (Not only coarse-text OWL) is an adaptation of OWL-ViT (NoctOWL) and OWLv2 (NoctOWLv2), designed for Fine-Grained Open-Vocabulary Detection (FG-OVD). Unlike standard open-vocabulary object detectors, which focus primarily on class-level recognition, NoctOWL enhances the ability to detect and distinguish fine-grained object attributes such as color, material, transparency, and pattern.

It maintains a balanced trade-off between fine- and coarse-grained detection, making it particularly effective in scenarios requiring detailed object descriptions.

You can find the original code to train and evaluate the model here.

Model Variants

• NoctOWL Base (lorebianchi98/NoctOWL-base-patch16)
• NoctOWLv2 Base (lorebianchi98/NoctOWLv2-base-patch16)
• NoctOWL Large (lorebianchi98/NoctOWL-large-patch14)
• NoctOWLv2 Large (lorebianchi98/NoctOWLv2-large-patch14)

Usage

Loading the Model

from transformers import OwlViTForObjectDetection, Owlv2ForObjectDetection, OwlViTProcessor, Owlv2Processor

# Load NoctOWL model
model = OwlViTForObjectDetection.from_pretrained("lorebianchi98/NoctOWL-base-patch16")
processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch16")

# Load NoctOWLv2 model
model_v2 = Owlv2ForObjectDetection.from_pretrained("lorebianchi98/NoctOWLv2-base-patch16")
processor_v2 = Owlv2Processor.from_pretrained("google/owlv2-base-patch16")

Inference Example

from PIL import Image
import torch

# Load image
image = Image.open("example.jpg")

# Define text prompts (fine-grained descriptions)
text_queries = ["a red patterned dress", "a dark brown wooden chair"]

# Process inputs
inputs = processor(images=image, text=text_queries, return_tensors="pt")

# Run inference
outputs = model(**inputs)

# Extract detected objects
logits = outputs.logits
boxes = outputs.pred_boxes

# Post-processing can be applied to visualize results

Results

We report the mean Average Precision (mAP) on the Fine-Grained Open-Vocabulary Detection (FG-OVD) benchmarks across different difficulty levels, as well as performance on rare classes from the LVIS dataset.