clip-vitb-mini-distilled

An 8.66M-parameter ViT image encoder (10.0% of a CLIP-B/16 image tower) producing 512-d embeddings compatible with the CLIP-B/16 LAION-2B text tower. The primary checkpoint was distilled on CC12M (10,968,539 images) against the generalized-Procrustes consensus of five CLIP teachers — never against the deployment teacher — and carries a frozen 512×512 rotation that maps its outputs into the deployment frame, where it outperforms the student distilled directly against that teacher on every task gauge, both seeds (full tables below).

Quick start (AutoModel)

import torch
from transformers import AutoModel, AutoImageProcessor

repo = "AbstractPhil/clip-vitb-mini-distilled"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained(repo)

px = proc(images=[img], return_tensors="pt")["pixel_values"]
emb = model.get_image_features(px)     # (1, 512), L2-normalized,
                                       # deployment (LAION-B/16) frame

The rotation is applied by default (config.apply_rotation); pass apply_rotation=False to get_image_features for the raw consensus-frame embedding. Weights are safetensors; the modeling code is in this repo (modeling_clip_mini.py, configuration_clip_mini.py).

Why a rotation exists at all — and when it doesn't. This model's consensus target is a generalized-Procrustes running mean of five teachers, which has no privileged frame: the student lands in the consensus frame rather than any teacher's, so a fitted rotation carries it the rest of the way. That is a property of how the target was built, not of consensus distillation. A sibling campaign built its consensus by mapping every teacher into the frame of one reference member of the set; its frame gap fell monotonically to ~.001 during training and it ships with no rotation. If you build your own consensus target, pick a reference member for a pinned frame or a mean for symmetry — and know which you chose.

Evaluation-path note: all published numbers were measured with the tensor-bicubic evaluation pipeline (interpolate to 160px, CLIP norm — the path implemented in loader.py and train/dist_bed.py). On CIFAR-scale (32px) inputs the shipped CLIPImageProcessor path yields 0.5545 vs 0.5708 zero-shot for the same weights, a resize-path difference that matters only for very small source images; verify against ledger numbers with the tensor path.

Repository inventory

Architecture

Results

Headline — champion (rotated) vs the directly-distilled student, CC12M, seed 0 / seed 1:

¹ Corrected 2026-10-08. Earlier versions of this table scored COCO retrieval over all 5,000 val images, but the rotation had been fitted on 2,500 of them, so half the scored images were the rotation's own fit set. These rows score only the other 2,500 images, in every column (a 2,500-image gallery, so they are not comparable with the 5,000-image figures elsewhere on this page). Scoring each half once with a rotation fitted on it and once with a rotation fitted on the other half puts the fit's own lift at +.033 / +.032 R@1 and +.047 / +.046 R@5 (seed 0 / seed 1). On held-out images the champion's retrieval lead over the InfoNCE student is within sampling noise at this gallery size: R@1 +.008 / +.006, R@5 +.009 / +.020. In these rows the InfoNCE student and the teacher use no fitted map; the InfoNCE student scored through a rotation fitted the same way has not been measured on retrieval. The superseded all-image figures (.1322 / .1360 and .3140 / .3300) reproduce exactly. The pure-MSE consensus row's rotated R@1 below (.046 / .051) was scored the same way and has not been re-scored. Raw results: ledgers/coco_heldout_rescore_2026-10-08.json and ledgers/coco_crossfit_2026-10-08.json.

Champion capability: 44.7% / 48.1% of the teacher's zero-shot mean (CIFAR-10+100) at 10% of its image-tower parameters. Scaling arc across data generations: 21.2% (118k images) → 42.0/41.5% (11M, direct InfoNCE) → 44.7/48.1% (11M, consensus + rotation).

Full CC12M matrix (nine objectives × two seeds, 88k steps ≈ 2.05 epochs; agree = cosine to deployment teacher unless (own) = the arm's own target):

Rotation forensics (agree→deployment pre → post; zs C10 pre → post; both seeds):

Note the pure-MSE consensus row: rotated agreement .765/.763 exceeds direct mimicry's in-frame .735/.738 — the consensus target carries the deployment teacher's geometry better than direct distillation preserves it — but without a contrastive term it cannot retrieve (rotated R@1 .046/.051). The composite's two terms are both load-bearing.

Generation one (COCO, 118k images, 8k steps), for the scale comparison: mimicry .2693/.2657 zs vs InfoNCE .2668/.2308 (mimicry won coarse zero-shot at small scale; the verdict reversed at 11M); InfoNCE retrieval .0580/.0544 vs .0248/.0262; consensus raw .1543/.1609 — its loss to single-teacher distillation was preregistered and refuted 2/2 at this scale before reversing at CC12M with the rotation. Full generation-one ledgers included.

Training

Everything needed to reproduce any checkpoint is in train/ (train/TRAINING.md is the step-by-step guide). Summary:

Evaluation protocol

CIFAR-10/100 zero-shot via the LAION-B/16 text tower ("a photo of a {class}"), images tensor-upscaled (bicubic) to the student's 160px input; teacher judged at its native 224px in the same harness; floor (random init) and ceiling (teacher) bracket every table. COCO retrieval: image→text R@1/R@5 against first-caption text embeddings, paired protocol: over the 5,000 val images for models with no fitted map (the matrix), and over the 2,500 val images the rotation was not fitted on for the rotated champion, with every comparison column re-scored on the same 2,500 (the headline table). Agreement: mean cosine to the arm's target embeddings on COCO val. Effective rank and a spread statistic (CV of Cayley-Menger 4-simplex volumes on a fixed 16-d projection) are read-only health instruments. Every evaluation is deterministic: --eval-only re-runs reproduce ledger rows to four decimal places, and this release was gated by fresh tokenless downloads reproducing the headline numbers exactly (raw-loader path and AutoModel path, bit-parity verified).

Limitations

Research model. 48% of teacher zero-shot at best; the preregistered 60% viability bar is unmet (the data-scaling curve is unbent — see the article). CIFAR absolute numbers are harness-scoped (32px upscale; see the evaluation-path note). Two seeds per objective; one batch size (256); one student size; one consensus family; one deployment frame. The rotation fit consumes 2,500 labeled pairs of the deployment frame; retrieval through the rotation is reported only on the 2,500 val images outside that fit set.

Related

• Campaign report: article_cc12m_distillation.md (in this repo)
• Teacher banks: bulk-cc12m-features · bulk-coco-features
• Memory-bank line: geolip-vit-captionbank-coco
• Objective genealogy: loss-manifest

References

• Radford et al., CLIP — https://arxiv.org/abs/2103.00020
• van den Oord et al., InfoNCE — https://arxiv.org/abs/1807.03748
• Yang et al., CLIP-KD — https://arxiv.org/abs/2307.12732
• Wu et al., TinyCLIP — https://arxiv.org/abs/2309.12314
• Zhai et al., SigLIP — https://arxiv.org/abs/2303.15343
• Gower, Generalized Procrustes Analysis — https://doi.org/10.1007/BF02291478
• Changpinyo et al., Conceptual 12M — https://arxiv.org/abs/2102.08981
• Cherti et al., OpenCLIP scaling — https://arxiv.org/abs/2212.07143