RF-DETR Small + text-aligned head (ONNX)

Open-vocabulary detection by adding one trained projection to a closed-set RF-DETR Small detector, served by VisionServe (Docker Hub).

The detector exports the 256-d feature f_i of each of its 300 object queries as an extra ONNX output, query_feats. A trained matrix P maps that feature into the text-embedding space of a CLIP or SigLIP text tower, and each class gets a cosine score against the embedding of its name:

logit_ic = a · ⟨ t̂_c , P f_i / ‖P f_i‖ ⟩ + b        conf = sigmoid(logit)

The vocabulary is the prompt, for example "cup. cola can. banana.". Changing it needs no re-export and no second detector: the text tower runs once per new vocabulary, and the result is cached. Boxes always come from the detector.

visionserve pull rfdetr-textalign-dec1-siglip      # also pulls siglip-text
visionserve run rfdetr-textalign-dec1-siglip img.png --prompt "cup. cola can. banana."

pull downloads one subfolder of this repo, writes a manifest.yaml, and pulls the text tower the model needs as a separate dependency.

Models

• dec1 is RF-DETR Small fine-tuned on the 22 tabletop classes with the backbone and box head frozen and only the last decoder layer trainable.
• dec1 held-out is the same recipe trained on 17 of the 22 classes. The other 5 names were held out of every trained component, so they can only be reached through a prompt.
• rfdetr-small-etri is the full fine-tune published as mtbui2010/rfdetr-small-etri-ONNX, re-exported with query_feats.

clip-text is the text half of CLIP ViT-B/32 (openai/clip-vit-base-patch32, MIT). It is needed by rfdetr-textalign-dec1 and rfdetr-textalign-etri and had no public ONNX export, so it is published here as its own catalog model. siglip-text comes from mtbui2010/siglip-base-patch16-224-ONNX.

The source repository also has -probe variants of three of these models (rfdetr-textalign-dec1-probe, -dec1-siglip-probe, -etri-probe). They are evaluation configs: the same files with conf_threshold: 0.001, so mAP is computed over the full precision/recall curve. They are not published separately. To get one, pull the model and lower postprocess.conf_threshold in its manifest.

How each model scores

The method request field picks how the head is applied (default when omitted: exact).

• exact: the formula above. It runs as an ONNX Runtime session (head.onnx) when the manifest names one, which every model here does.
• folded: drops ‖P f_i‖, which makes the head a single [C, 256] matrix.
• gated: the detector's own class head gives the score used for thresholding and ranking, and the text-aligned head gives only the name. Changing the vocabulary then changes labels and leaves confidences unchanged.

Each manifest's conf_threshold was tuned for one method. Changing the method changes the scale of the thresholded score.

• rfdetr-textalign-etri: 0.28 was tuned for exact, the default.
• rfdetr-textalign-dec1 and -dec1-siglip: 0.35 was tuned for gated, which is how they are served (--method gated, or -F method=gated on /api/predict). With the default exact, the thresholded score is the text-aligned head's, whose scale is different, and 0.35 returns few or no detections.

visionserve run rfdetr-textalign-dec1-siglip img.png --prompt "cup. cola can." --method gated

Files

dec1/detector.onnx and dec1-siglip-prod/detector.onnx are the same file (same sha256). Both folders carry it so that each folder is complete on its own.

sha256

visionserve pull checks every file it downloads against these digests, and the generated manifest pins them again so the server re-checks them at load.

How head.onnx is made

head.onnx holds no new weights. It is proj.bin written as an ONNX graph by models/rfdetr-textalign-dec1-siglip/export_head_onnx.py, so that method exact runs on ONNX Runtime instead of in Go. The export is deterministic. Re-running it on each proj.bin here, with onnx 1.21.0, gives byte-identical files:

python3 models/rfdetr-textalign-dec1-siglip/export_head_onnx.py \
    --proj <folder>/proj.bin --out <folder>/head.onnx

The script also checks the graph against a NumPy reference (max |Δlogit| about 2e-6 on CPU). The server compares one query per request against proj.bin and refuses a head.onnx that disagrees. If you swap in another projection, for example proj-openvocab.bin renamed to proj.bin, re-export head.onnx too. The manifests set runtime.threads: {head: 1}. The head takes about 1 ms on one thread, and ORT's default thread pool for it slowed the detector about 3x on CPU.

Training and data

• Detector base. RF-DETR Small, COCO-pretrained, from Roboflow (Apache-2.0).
• Fine-tuning data. tabletop-22: 247 usable images of household and tabletop objects on an indoor work surface, 1447 boxes, 22 classes. The authors captured it with a robot arm's wrist camera at the Korea Electronics Technology Institute (KETI). It contains no people and no third-party imagery, and the authors release it under Apache-2.0. During development it was called ETRI, which is where the -etri model names come from. Every model uses the same split: 185 training and 62 held-out images, by image, seed 0.
• Projection P. Trained by distillation from the text tower's teacher. The local READMEs describe the recipes:
  • etri: CLIP text anchors on the 22 class names, plus label-free CLIP region distillation over the detector's own proposals. It was trained with the (‖P f‖−1)² penalty (flags bit 0 set), with a = 13.7738 and b = −5.0417.
  • dec1: CLIP teacher, train_qf.py --n-coco 30000 --steps 6240 --lam-dec 4 --lam-norm 1, over COCO train2017 images used as an unlabelled distillation pool. No norm penalty (flags 0).
  • dec1-siglip: SigLIP teacher, a mixed pool of in-domain tabletop crops and 2000 COCO images with proportional weighting. proj-openvocab.bin uses 5000 COCO images with balanced weighting.
  • dec1-siglip-prod: a SigLIP-space projection (d_text 768) for the 22-class dec1 detector. The source repository does not document its recipe or any measurement for it.

Measured results

Every number below comes from the VisionServe source READMEs named in each line. All are on the 62 held-out tabletop-22 images unless a line says otherwise, and all were measured through the VisionServe server.

rfdetr-textalign-etri (README). RTX A6000, ONNX Runtime 1.26, CUDA EP:

These p50 values were measured before head.onnx existed, with exact running in Go.

rfdetr-textalign-dec1 (README). COCO-style mAP, measured through the -probe config (conf_threshold 0.001):

conf_threshold 0.35 is the F1 peak (0.768) for gated on the same images.

rfdetr-textalign-dec1-siglip (README). This table is naming accuracy (top-1) on detections matched to a ground-truth box. It is not mAP. "Unseen 5" means the 5 names that were held out of every trained component:

No end-to-end mAP is documented for this model. Its conf_threshold (0.35) was taken from the 22-class sibling and has not been re-tuned.

rfdetr-textalign-dec1-siglip-prod: no measurement is documented.

Limitations

• One domain. The detectors were fine-tuned on one indoor tabletop scene. The etri head does not transfer: on COCO val2017 novel categories it scores 3.7% box-conditioned top-1, against 57.0% for a CLIP crop classifier. The dec1-siglip head was distilled on a mixed pool to address this. Its COCO numbers are in the table above.
• Near-synonyms. The head cannot separate names that the text tower barely separates, such as coffee and coffee can (cosine 0.926 in CLIP text space). Four of the 22 classes score 0% with the etri head for this reason.
• Closed choice. The prompt is a closed list. Every kept box is given the nearest name in it, so a box whose real name is missing from the prompt still gets a label.
• Thresholds are per method. See How each model scores. rfdetr-textalign-dec1-siglip-prod ships with conf_threshold 0.001, so it returns many low-confidence detections. Threshold them on the client, or raise the value in the manifest.
• Resampling gap. P was fitted on features from Pillow's bilinear resize, and the server uses disintegration/imaging. Measured on the etri head: median |Δconf| 0.0027, max 0.112.

License

Apache-2.0 for the detectors, projections and heads. They are fine-tunes of Roboflow RF-DETR (Apache-2.0), plus projections trained by the authors. The text towers they depend on are also permissive: SigLIP base-patch16-224 is Apache-2.0, and CLIP ViT-B/32 is MIT. The clip-text/ folder is redistributed under the MIT licence in clip-text/LICENSE (Copyright (c) 2021 OpenAI). No AGPL component is used anywhere.