UI element detector (RF-DETR-Medium, 1280 px)
Finds the parts of a web-page screenshot and sorts them into five classes:
| id | class | meaning |
|---|---|---|
| 0 | icon |
a pictogram or other control with no readable text (icon buttons, checkboxes, radios, swatches) |
| 1 | text_control |
a link or button with text on it |
| 2 | input |
a text field, select or text area |
| 3 | text |
a block of plain text (paragraph, heading, label) |
| 4 | image |
a picture |
It is RF-DETR-Medium fine-tuned for 33 minutes on one GPU on 1,274 screenshots whose labels were read from the page structure, not drawn by hand. Dataset and labelling scripts: oneryalcin/ui-controls-web.
Changed on 2026-11-20: the file ui-controls-rfdetr-medium-1280.pth was replaced. The first version had three
classes (ids 0 to 2 only) and is still in this repository's history. Code that kept only ids 0 to 2 still works,
but small text links now often come out as text (id 3), so keep id 3 as well if you want everything clickable.
Use
# pip install rfdetr pillow
from PIL import Image
from rfdetr import RFDETRMedium
model = RFDETRMedium(pretrain_weights="ui-controls-rfdetr-medium-1280.pth", resolution=1280)
det = model.predict(Image.open("screenshot.png").convert("RGB"), threshold=0.2)
names = ["icon", "text_control", "input", "text", "image"]
for box, cls, score in zip(det.xyxy, det.class_id, det.confidence):
print(names[cls], [round(v) for v in box], round(float(score), 2))
Trained with rfdetr==1.12.1. Use threshold 0.2; lower it to 0.1 or 0.05 for more boxes.
Results
52 held-out screenshots from 11 sites not seen in training, 1440×900: 2,866 labels, of which 1,911 are clickable controls. A label counts as found when a box overlaps it with IoU ≥ 0.5 or the two centres lie inside each other. "Small" means under 24 px on the short side.
| Threshold 0.2 | UI-DETR-1 | First version (three classes) | This model |
|---|---|---|---|
| Clickable controls found, by a box of any class | 87.1% | 91.5% | 94.5% |
| Clickable controls found, by boxes of the three control classes only | 91.5% (3,284 boxes) | 90.9% (3,221 boxes) | |
| Small clickable icons found | 57/67 | 56/67 | 56/67 |
| Small text links found | 262/283 | 171/283 | 221/283 |
| All labels found, plain text and pictures included | 89.4% | not trained for it | 93.9% |
| Boxes on the 52 pages | 5,793 | 3,284 | 4,571 |
| Its boxes that match a label | 71% | 78% | |
| Classes | one | three | five |
| Weights | 535 MB | 142 MB | 142 MB |
- As a controls detector it is level with the first version, not better. The 94.5% counts boxes the model calls
textorimagethat land on a control, and comes with about 40% more boxes. What is new is plain text and pictures. - Right class for 89.6% of all labels at threshold 0.2 (plain text 809 of 881, large text controls 836 of 872).
- Lower thresholds: 96.5% of all labels found at 0.1 (7,643 boxes) and 97.0% at 0.05 (12,188 boxes).
- Speed: the first version took 0.30 s per image on an Apple M-series laptop against 0.57 s for UI-DETR-1. This version is the same network and was not timed again.
Limits
- Web pages only, 1440×900, scale 1. Not tested on native desktop apps, phones, or other sizes.
- Links against plain text. Of 283 small text links, 221 are boxed but only 171 are called
text_control; most of the rest come out astext. A link and plain text often look the same. - Weak spots: small pictograms that cannot be clicked (3 of 26 found), and
image(26 of 41 with the right class; only 382 training labels). - No gain in end-to-end clicking. In a 62-target "find the control for this goal" test on five other sites, a
pipeline that names this model's
iconboxes scored 51/62 at threshold 0.2 and 54/62 at 0.1. The first version scored 54/62 (twice) and UI-DETR-1 55/62. For that use, 0.1 is the better threshold. - Labels are automatic and the classes are heuristic; see the dataset card.
- One training run, with thresholds compared on the test pages themselves. No repeat runs.
Licence
Apache 2.0, the same as the RF-DETR-Medium weights it starts from.
The file is a PyTorch Lightning checkpoint (a pickle) as written by rfdetr; it holds weights and training settings only.