Single-Pass GUI Action Prediction
Examples · Results · Quick start · 简体中文
GUI-Swift-4B turns a screenshot, an instruction and action history into one desktop GUI action through a single native causal forward over 131 fixed value slots. It is a lightweight LoRA release built on GUI-Owl-1.5-4B-Instruct, with the original vocabulary head retained and no separate autoregressive text writer.
Release: Desktop131 SFT · step3368 / Adapter: 33.03M parameters · 132 MB
This is a desktop research checkpoint. “Single-pass” describes the inference mechanism; a matched speed or task-success advantage has not been established.
Real validation examples
These are actual saved predictions from this checkpoint's 1,967-row AgentNet development validation set. The gallery is selected after evaluation and includes a text failure. Screenshots show the current input frame; action history was also part of the model input. These are next-action examples, not completed-task demonstrations.
GIMP · Find the Desktop folderACTION CORRECT · OFFSET (2, 0) / 1000 Input screenshot · click for full size Model prediction Reference action Please use GIMP to open the "MMQ.png" image on your desktop, add the black text "sky" at the top of the image with a font size of 100 and make it bold. Historical action records: 4 · Strict action match: false | Writer · Enter a search termEXACT ACTION MATCH Input screenshot · click for full size Model prediction Reference action Help me open the desktop document "144.odt", change the overall font size of the document to 16pt and the font color to green, use the find and replace function to replace "CAGR" with "WWAA", and then save it as "yrrte7" on the desktop after completion. Historical action records: 8 · Strict action match: true |
Impress · Insert a slideEXACT ACTION MATCH Input screenshot · click for full size Model prediction Reference action In Impress,add two new slides.Enter“Ancient History”and“Modern Art”in the title areas of slides 1 and 2 respectively.Enter the table of contents“・Egyptian・Greek・Roman”in the content area of slide 3.After completion,change the font to“Likhan”uniformly. Historical action records: 1 · Strict action match: true | Calc · Rename a worksheetEXACT ACTION MATCH Input screenshot · click for full size Model prediction Reference action Create three worksheets named "5, 6, 7" and change the color labels of worksheet 567 to "red, yellow, blue", then hide them. Historical action records: 12 · Strict action match: true |
A limitation · Repeated text
ACTION CORRECT · TEXT MISMATCH
Input screenshot · click for full size
Model prediction{"action": "type", "text": "・ Egyptianian GreekGreek Roman Roman"}
Reference action{"action": "type", "text": "・Egyptian・Greek・Roman"}
In Impress,add two new slides.Enter“Ancient History”and“Modern Art”in the title areas of slides 1 and 2 respectively.Enter the table of contents“・Egyptian・Greek・Roman”in the content area of slide 3.After completion,change the font to“Likhan”uniformly.
Historical action records: 13 · Strict action match: false
Saved prediction & provenance (JSON)
Inspect the GIMP click locationsThe model predicts (155, 243) and the reference is (153, 243), on a 0–1000
coordinate scale. The action type agrees and the coordinate offset is (2, 0);
the strict full-action match is false. The circles/cross are presentation
overlays, not model outputs or evidence of executing the click.
Source: AgentNet / OpenCUA. Example manifest · Original screenshot license. The raw validation corpus is not bundled.
Results at a glance
| Saved evaluation | Result | What it measures |
|---|---|---|
| OSWorld-derived selected suite | 16.29% · 58 / 356 | Environment-verified full task success among scored tasks |
| Development validation · action type | 75.60% · 1,487 / 1,967 | Whether the action type agrees |
| Development validation · position @50/1000 | 49.64% · 548 / 1,104 | Action type agrees and both x/y errors are <= 50 on a 0–1000 scale |
| Development validation · text | 30.08% · 120 / 399 | Exact text action match on reference type rows |
The position threshold was changed from 20 to 50 after inspecting the results, then recounted on the same saved predictions. It allows 5% of each screen dimension and measures the primary point, not widget hit rate or complete action correctness. The original @20 result is 41.30% (456/1,104); strict full-action matching is 24.05% (473/1,967). Metric definitions and original results.
The OSWorld-derived run selected 361 of 369 tasks: 356 scored, 5 unscored; mean reward among scored tasks was 0.1712077. This is not a complete official leaderboard score. Validation was repeatedly used during development, and this checkpoint was chosen after observing results. No new model inference or environment run was performed; the five examples above do not estimate average performance.
Next direction — reinforcement learning: use this SFT checkpoint as a starting point and train with environment-verifier feedback to improve end-to-end task completion. The 16.29% is the current measured SFT result; improvement through RL is a future objective and has not been established for this release.
Original saved metrics · Position recount · Training and evaluation details
Quick start
Use Python 3.10+ and an NVIDIA GPU. Download the adapter repository, then prepare the pinned backbone and GUI-Decisions dependency on a disk with enough space.
hf download dongjintao/GUI-Swift-4B --local-dir GUI-Swift-4B
cd GUI-Swift-4B
python -m pip install -r requirements.txt
python prepare_dependencies.py --destination /path/on/data-disk/gui-swift-deps
python infer.py \
--base-model /path/on/data-disk/gui-swift-deps/GUI-Owl-1.5-4B-Instruct \
--gui-decisions /path/on/data-disk/gui-swift-deps/GUI-Decisions \
--request examples/request.json
The base download is approximately 8.9 GB. Matching existing local dependencies can be used instead. The script predicts an action; it does not operate your desktop. The bundled runnable request is a synthetic smoke example, separate from the saved validation gallery. Gallery JSON files document historical results and are not complete replay requests because historical images/actions are omitted.
Request format and loading notes{"instruction": "Click the Save button.",
"image": "screen.png", "history": []}Image paths are relative to the request file. History records contain image and
actions (actually executed canonical actions); the original runtime retains all
supplied actions and the last two historical screenshots. Do not put the reference
next action into the input. Use the custom infer.py entry point: a generic
generate() call or attaching PEFT to the entire VLM does not reproduce the protocol.
The frozen loader attaches LoRA to the language-model submodule.
Design & credits
We thank GUI-Decisions for its
256-bin coordinate representation and readout, our principal inspiration from
that work. Single-forward complete-action prediction was already being explored
in this project. The runtime also directly reuses the pinned upstream slotread.train.read
and label/numerical utilities; design history and implementation reuse are credited separately.
We also thank GUI-Owl / Mobile-Agent for the backbone, OpenCUA / AgentNet for the validation examples, and PEFT / Transformers for adapter and model support. The external GUI-Decisions snapshot has no license file and is not bundled or relicensed here. Third-party sources and terms.
Research notes
- Current strengths and gaps: the saved cases show simple click, key and text prediction. Text repetition, dragging, misclicks and premature stopping remain limitations. No verified mobile compatibility or general latency advantage is claimed.
- Scope: SFT, desktop-only, 131 slots. This release is not Unified133 or the later RL model. Terra-assisted trajectories are not standalone model results.
- Reproducibility: 504 exported adapter tensors matched the checkpoint exactly. Separate-process checks matched inputs, logits and decoded actions on two saved inputs. One RTX 4090 used about 9.1 GiB peak allocated memory in that bounded check; this is not a maximum-context memory bound.
Full protocol and training recipe · Release verification · Provenance · File checksums