Muse-Glimmer-30B-GUI-Grounding-Fast

A lightweight LoRA adapter for Meta's Muse-Glimmer-30B that turns it into a fast, reliable GUI-grounding model: given a screenshot and an instruction, it outputs the target element's bounding box directly as [x1, y1, x2, y2] — no verbose reasoning.

Built for computer-use / GUI agents that need an exact click coordinate now, at a fraction of the base model's inference cost.

Why this exists

The base Muse-Glimmer-30B already locates UI elements well, but as an agentic model it "thinks out loud" — it buries the coordinate under ~200 tokens of reasoning and only emits a cleanly parseable box part of the time. That's expensive and unreliable inside an agent loop.

This adapter teaches it to answer with only the coordinate.

Before vs. after (real example)

Instruction: "close this window" on a 960×540 screenshot (ground-truth box [0.948, 0.144, 0.994, 0.207]).

Base Muse-Glimmer-30B (rambles, coordinate buried in reasoning, often never emits a clean box):

 to=self ...Close this window. The window has minimize, maximize, close at top
right. Close is X. Coordinates. Approx top right... x ~ 0.96 to 0.99, y ~ 0.17
to 0.21. Let's give [958, 170, 985, 205] but need relative 0-1. So maybe
[0.958, 0.168, 0.985, 0.210]. Let's approximate. Better estimate: ...

(~200 tokens, and the parser only sometimes finds a usable box)

This adapter (one clean answer, ~5 tokens):

[0.958, 0.168, 0.985, 0.210]

Same location — but deterministic, instantly parseable, and ~10× cheaper per action.

What's novel here

• Reliability, not just capability. Rather than chasing raw grounding accuracy, we target the output-reliability gap that makes strong VLMs hard to deploy as agents: we lift the usable-answer rate from 63% → 99.8%.
• Diversity beats steps. Our first attempt (desktop-only, ~5k unique images) overfit and stalled at 70.5%. Swapping in ~40k unique screenshots (desktop + mobile) — not more training steps — jumped accuracy to 87% and fixed the weak domains. The lesson: for grounding LoRAs, image diversity is the dominant lever.
• Tiny, surgical, cheap. A single-node LoRA touching 0.7% of params (vision tower frozen) converts a general agentic VLM into a specialized, production-ready GUI grounder — no full fine-tune, no RLHF.
• First GUI-grounding adapter for Meta's Muse-Glimmer-30B (to our knowledge), fully reproducible on open weights + open data.

Results

Headline — full ScreenSpot-v2 (all 1,272 samples)

Metric: predicted bounding-box center falls inside the ground-truth box (standard ScreenSpot click accuracy).

The reliability win (diagnostic, 400-sample subset)

Under a plain "output the box" prompt, the base model reasons instead of emitting a clean coordinate:

The 39.5% base figure is harness-limited — with its own agentic harness the base grounds at ~75%. Our adapter makes that ability come out reliably (parse 63% → ~100%) and directly.

By platform & target type (full benchmark)

GitLab 93.2 · macOS 91.8 · iOS 90.3 · forum 88.6 · web/shop 88.2 · Android 87.2 · tool 86.7 · Windows 81.1 | text 94.2 · icon 80.3

How we got here — diversity beats steps

Run-1 (desktop-only, ~5k unique images) overfit and stalled at 70.5%. Run-2 (+mobile, ~40k unique images) reached 88.1% — the gain came from image diversity, not more training steps.

Context vs other GUI-grounding models (ScreenSpot-v2)

We land above OS-Atlas-7B and Qwen2-VL, competitive with UGround-V1, just behind UI-TARS-7B — with a lightweight LoRA on a general-purpose model. Other models' scores are published full-benchmark numbers; ours is our own eval harness on the full 1,272-sample set with the standard center-in-box metric — reproduce it with the code below. Not an officially verified leaderboard submission.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForMultimodalLM
from peft import PeftModel

BASE = "meta-models/Muse-Glimmer-30B"
ADAPTER = "lemuralabs/Muse-Glimmer-30B-GUI-Grounding-Fast"

proc = AutoProcessor.from_pretrained(BASE)
model = AutoModelForMultimodalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER).eval()

img = Image.open("screenshot.png").convert("RGB")
instruction = "the settings gear icon"
prompt = (f'In this UI screenshot, locate the element described as: "{instruction}". '
          'Respond with ONLY its bounding box as [x1,y1,x2,y2], each a 0-1 float '
          'relative to image width/height.')

msgs = [{"role": "user", "content": [{"type": "image", "image": img},
                                     {"type": "text", "text": prompt}]}]
inp = proc.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                               return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inp, max_new_tokens=32, do_sample=False)
print(proc.tokenizer.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True))
# -> e.g. [0.912, 0.043, 0.958, 0.089]

Coordinates are normalized 0–1 (relative to image width/height). Take the box center as the click point.

Training

• Method: LoRA (rank 32, α 64, dropout 0.05) on the language decoder only (model.language_model.*); vision encoder frozen.
• Trainable params: ~210M (0.7% of the 30B model).
• Data: ~160k GUI-grounding examples across ~40k unique screenshots, from the open OS-Atlas corpus (desktop + mobile domains).
• Schedule: lr 5e-5, cosine, bf16, gradient checkpointing; best checkpoint at ~500 steps (early stopping — later steps overfit).

Limitations

• Icons and dense/small targets remain the hardest (80% vs 94% for text).
• Trained/evaluated on English UI screenshots.
• Windows is the weakest platform (81%).
• Scores are from our own eval harness (full 1,272-sample ScreenSpot-v2, standard center-in-box metric), not an officially verified leaderboard submission.
• Requires transformers >= 5.15 (base model architecture requirement).

License & attribution

Released under Apache 2.0. This is a derivative adapter built on Meta's Muse-Glimmer-30B (Apache 2.0) and the open OS-Atlas dataset. Use is subject to the base model's Usage Policy. Credit to Meta for the base model.