SmolVLM-256M-GUI-Grounding

A 256M-parameter vision-language model fine-tuned for GUI grounding: given a screenshot and a natural-language instruction ("click the export button"), it returns the point to click.

It is a fine-tune of SmolVLM-256M-Instruct, about 0.5 GB of weights, small enough for laptops and on-device use.

A larger sibling trained with the same recipe is avinash2468/SmolVLM2-500M-GUI-Grounding.

Coming soon: training and evaluation code, data details, and a technical report. Follow avinash2468 on Hugging Face or get in touch to hear when they are out.

Results

Accuracy (%) — a prediction counts as correct when the predicted point lies inside the target element's box.

• 1 pass is a single forward pass on the full screenshot — exactly what the quick start below does.
• 4 / 12 passes use a test-time refinement procedure that re-examines the screen before answering. It is not included in this release; those rows are reported for reference.
• Extra passes matter most on high-resolution professional screens (ScreenSpot-Pro: 14.3 → 40.0) and little on ScreenSpot-v2, where a single pass already reads the screen well.
• Twelve passes of a 256M model cost roughly as much compute as one pass of a 3B model, so the 12-pass numbers should be compared with ~3B models at similar compute, not read as a free efficiency gain.

Examples

Hand-picked single-pass predictions from this model on benchmark screenshots, chosen to show instructions that need some reading or reasoning. They are illustrations, not a random sample — the table above is the real measure. In each image the red dot is the model's click, the green box is the ground-truth element, and the right panel zooms into the yellow area.

1. “Click the B21 cell” — Spreadsheet (OSWorld-G): the model has to work out column B and row 21 itself; the cell holds a name, not the text “B21”.

2. “Three-dot menu icon to the right of 'Taylor Swift - Shake It Off' entry” — Browser history (OSWorld-G): one tiny icon out of a dozen identical ones, picked out by the row it belongs to.

3. “Turn on the alarm for 3:15” — Phone clock app (ScreenSpot-v2): the toggle on the 03:15 row, not one of the other alarms.

4. “switch map layer to the opnvkarte” — OpenStreetMap (ScreenSpot-v2): matches a loosely spelled instruction to the “ÖPNVKarte” layer.

5. “Click on the question icon of the "save" dialog.” — LibreOffice (OSWorld-G): a small icon inside a dialog stacked on top of two other windows.

Screenshots are from the ScreenSpot-v2 and OSWorld-G benchmarks.

Model details

Quick start

Requires transformers >= 5.0, torch, pillow.

import math, re, torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText

repo = "avinash2468/SmolVLM-256M-GUI-Grounding"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, torch_dtype=torch.bfloat16).to(device).eval()

SYSTEM = ("You are a GUI grounding agent. You are given a screenshot and an instruction describing a single UI "
          "element to interact with. Identify the location of that element.")
PROMPT = ("Instruction: {instruction}\n\n"
          "Respond with ONLY the click location as coordinates in the format (x, y), where x and y are integers "
          "from 0 to 100 giving the horizontal and vertical position as a fraction of the image width and height "
          "times 100. Do not output anything else.")

def prepare(img):
    """Resize so the long side is a multiple of 512 px, clamped to 2048-2560 (the resolution used in training)."""
    w, h = img.size
    target = min(max(512 * math.ceil(max(w, h) / 512), 2048), 2560)
    s = target / max(w, h)
    return img.resize((round(w * s), round(h * s)), Image.LANCZOS)

def ground(image_path, instruction):
    image = Image.open(image_path).convert("RGB")
    view = prepare(image)
    messages = [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
        {"role": "user", "content": [{"type": "image"},
                                     {"type": "text", "text": PROMPT.format(instruction=instruction)}]},
    ]
    text = processor.apply_chat_template(messages, add_generation_prompt=True)
    inputs = processor(text=[text], images=[[view]], return_tensors="pt",
                       images_kwargs={"size": {"longest_edge": max(view.size)}})
    inputs = inputs.to(device, dtype=torch.bfloat16)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
    answer = processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
    pairs = re.findall(r"(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)", answer)
    if not pairs:
        return None
    x, y = (float(v) for v in pairs[-1])
    return x / 100 * image.width, y / 100 * image.height  # pixel coordinates in the original screenshot

print(ground("screenshot.png", "Click the Save button"))

Evaluation protocol

• Benchmarks: ScreenSpot-v2 (mobile, web, desktop; 1,272 items), ScreenSpot-Pro (high-resolution professional desktop apps; 1,581 items), OSWorld-G (Linux desktop; the 510 items with a ground-truth box — the remaining refusal items are excluded).
• Metric: predicted point inside the ground-truth box; an unparseable answer counts as a miss.
• Decoding: greedy, batch size 32, bf16. Greedy outputs vary slightly with batch size and padding, so expect small (≲ 0.5 point) differences when re-running.
• Checkpoint: the final checkpoint of training; no checkpoint selection on the test sets. Single training seed; no error bars.
• Contamination: exact-duplicate image checks found no overlap between the training data and the three benchmarks.

Limitations

• Icons are much harder than text. Most remaining errors are choosing the wrong one among visually similar adjacent icons.
• Very wide or very high-resolution screens (ultra-wide monitors, small targets on 4K displays) remain hard in a single pass.
• Not state of the art. Larger (2–3B+) grounding models score substantially higher; the point of this model is its size.
• Grounding only. It predicts a single click point for a single-element instruction; it does not plan multi-step tasks, and it is not trained to refuse when the element is absent.
• Latency and memory on CPU / mobile hardware have not been measured yet.
• As with any GUI agent component, do not let it act on real systems without human oversight.

License

Apache-2.0, the same as the base model. The training data are publicly released datasets, each under its own license; please check those licenses for your use case.

Contact

Built by Avinash Prabhu Puppala. I'm open to research collaboration, compute partnerships and research roles on small / on-device GUI agents.

• Email: avinashprabhu2001@gmail.com
• LinkedIn: linkedin.com/in/avinash-prabhu-p

Citation

@misc{smolvlm256m_gui_grounding_2026,
  title        = {SmolVLM-256M-GUI-Grounding: a 256M-parameter GUI grounding model},
  author       = {Puppala, Avinash Prabhu},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/avinash2468/SmolVLM-256M-GUI-Grounding}}
}