bobby-control-detector

Finds the interactable controls in a web page screenshot: buttons, links, text fields, search boxes, selects, checkboxes, radio buttons, switches and tabs, all as one interactable class. It is a fine-tune of PekingU/rtdetr_v2_r18vd (RT-DETRv2 with a ResNet-18 backbone) on synthetic pages labeled from the DOM, made for the bobby-browser automation runtime.

• Fine-tuned from the base model at revision 5650961749fa93567c0d46fc7f43ea4f9e914107.
• Not affiliated with or endorsed by the RT-DETR authors.
• Trained by Sasan Sotoodehfar, CAVI AI (https://cavi-ai.xyz).

Contents

• One class: interactable, id 0. The model finds controls; it does not name them or tell their role. Pair its boxes with OCR or the page's DOM for that.
• Input: the RGB screenshot stretched to 1024×544 (width × height) with bilinear resampling and scaled to 0–1, without mean/std normalization. preprocessor_config.json does this for transformers.
• ONNX input pixel_values: [1, 3, 544, 1024]. Outputs logits: [1, 300, 1] and pred_boxes: [1, 300, 4] as center x, center y, width, height, normalized to the image. A box's score is the sigmoid of its logit. No NMS is needed.
• Use a score threshold of 0.6, the threshold the results below were measured at.

Requirements

• transformers: pip install transformers torch pillow
• ONNX: pip install onnxruntime numpy pillow huggingface_hub
• Tested with Python 3.12, torch 2.15.0, transformers 5.22.1 and onnxruntime 1.31.1.

Usage

transformers

from transformers import pipeline

detector = pipeline("object-detection", model="cavi-ai/bobby-control-detector", threshold=0.6)
for hit in detector("screenshot.png"):
    print(round(hit["score"], 3), hit["box"])

• box holds xmin, ymin, xmax, ymax in screenshot pixels, rounded to whole pixels.

ONNX Runtime

import numpy as np
import onnxruntime
from huggingface_hub import hf_hub_download
from PIL import Image

session = onnxruntime.InferenceSession(hf_hub_download("cavi-ai/bobby-control-detector", "onnx/model.onnx"))
image = Image.open("screenshot.png").convert("RGB")
width, height = image.size
pixels = np.asarray(image.resize((1024, 544), Image.BILINEAR), dtype=np.float32) / 255.0
logits, boxes = session.run(["logits", "pred_boxes"], {"pixel_values": pixels.transpose(2, 0, 1)[None]})
scores = 1 / (1 + np.exp(-logits[0, :, 0]))
for score, (cx, cy, w, h) in zip(scores, boxes[0]):
    if score >= 0.6:
        print(round(float(score), 3), [round(float(v), 1) for v in ((cx - w / 2) * width, (cy - h / 2) * height, w * width, h * height)])

• Each box prints as x, y, width, height in screenshot pixels.

Training data

• 3,000 synthetic pages written by generate_pages.py: navigation bars, forms, cards, tables, toolbars, tabs, result lists, filters, banners, prose and pagers, with random light and dark themes, fonts, sizes, corner radii and spacing.
• Look-alikes that are not controls (status badges, table header cells, plain text) are left unlabeled, so the model learns to skip them.
• Each page was rendered in headless Chromium at 1900×980 with a device pixel ratio of 1, and the box of every visible interactive element was read from the DOM: 52,718 controls (22,884 links, 19,008 buttons, 3,441 tabs, 2,332 checkboxes, 2,269 text fields, 1,089 radio buttons, 858 search boxes, 498 selects, 339 switches). Boxes with a side under 4 pixels were dropped.
• No real-site screenshots and no personal data.

Training

• Round 1: 1,500 pages (1,350 train, 150 validation), fine-tuned from the base model at a learning rate of 1e-4 (1e-5 for the backbone) on a one-cycle schedule planned for 30 epochs, stopped after 9.
• Round 2: 3,000 pages (2,700 train, 300 validation), continued from round 1's best checkpoint at a learning rate of 5e-5 (5e-6 for the backbone) on a one-cycle schedule over 10 epochs. The released checkpoint is epoch 7, the best by validation F1.
• Both rounds: AdamW, gradient clipping at 0.1, batch size 4, input 1024×544, the best checkpoint kept by validation F1.
• About 50 minutes in total on an Apple M5 Max GPU with PyTorch MPS.
• train_detector.py reproduces the training and eval_detector.py the evaluation and the ONNX export.

Measured results

Hardware: Apple M5 Max.

• A detection counts when it overlaps a labeled control at IoU 0.5 or more, one to one, at a score of 0.6 or more.
• The 300 held-out pages were in neither round's training set.
• Of the 46 extra boxes on the hand-built pages, 15 are on an image-grid verification page whose ground truth lists no controls, and 6 are on controls behind an open modal dialog, which the ground truth leaves out.
• The page generator was extended after round 1 missed controls on the hand-built pages (result lists, bare checkboxes, header search fields, table headers), so those pages are not a fully independent test.

• The ONNX export matches PyTorch within 2.3e-5 on both outputs for a random input.
• With the preprocessing above in Rust and tract, a test screenshot gives the same 24 boxes as PyTorch, with scores within 0.0005.

Limitations

• Trained on synthetic desktop pages at 1900×980 only. Other aspect ratios are stretched to the input size, and mobile layouts are untested.
• Custom-drawn controls, such as canvas buttons, are found only when they look like common web controls.
• It finds whatever is visible, including controls behind a translucent overlay, and knows nothing about whether a control is enabled.

License

• Model weights and configuration files: Apache-2.0, as the base model. See LICENSE.
• Training and evaluation code in bobby-browser: MIT.

Citation

The base model:

@misc{lv2024rtdetrv2,
  title         = {RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer},
  author        = {Wenyu Lv and Yian Zhao and Qinyao Chang and Kui Huang and Guanzhong Wang and Yi Liu},
  year          = {2024},
  eprint        = {2407.17140},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}

Links

• Base model: https://huggingface.co/PekingU/rtdetr_v2_r18vd
• Training code: https://github.com/cavi-ai/bobby-browser/tree/main/scripts/perception
• bobby-browser: https://github.com/cavi-ai/bobby-browser