CLIP ViT-B/32 — Fine-Grained Visual Product Retrieval

A CLIP ViT-B/32 image encoder (initialised from laion2b_s34b_b79k) fine-tuned for fine-grained image retrieval across three diverse domains — fashion products, pet breeds, and general objects — with a single shared encoder.

What it does

Given an image, it produces a 512-dim L2-normalised embedding. Cosine similarity between embeddings retrieves same-category items, specialised for fine-grained distinctions (e.g. Oxfords vs. Sneakers, Persian vs. Ragdoll).

Results (held-out test splits — no leakage)

Fine-tuned on the train split of each dataset, evaluated on the held-out test split. Retrieval mean Average Precision at the fine-grained label level:

Precision@1 also improved on all three (Zappos 0.836→0.868, Pets 0.793→0.867, Caltech 0.908→0.955).

Training

• Objective: InfoNCE over same-label image pairs (in-batch negatives), temperature 0.05.
• Data: 52,959 training images pooled across the three datasets, labels namespaced per dataset so they never collide.
• Schedule: 2 epochs, batch 128 pairs, AdamW lr 1e-5, mixed precision, ~7 min on a single 24 GB GPU.

Usage (open_clip)

import open_clip, torch
from huggingface_hub import hf_hub_download

model, _, preprocess = open_clip.create_model_and_transforms(
    "hf-hub:vivekkopthsd/clip-vitb32-product-retrieval")
model.eval()
img = preprocess(Image.open("shoe.jpg")).unsqueeze(0)
with torch.no_grad():
    emb = torch.nn.functional.normalize(model.encode_image(img), dim=-1)

CPU deployment (ONNX)

An ONNX export of the image encoder (clip_vitb32_finetuned_image.onnx, opset 17, dynamic batch, L2-norm fused) is included — numerically identical to PyTorch (max abs diff 4e-07) and runs at ~37 ms/image on CPU via onnxruntime.

Limitations

• Fine-tuned on three specific datasets; retrieval quality on unrelated domains is not guaranteed and may have regressed from the general-purpose CLIP baseline.
• Image encoder only — the text tower is unchanged and not part of this fine-tune.