::: {align="center"}

AgentEye Grounder --- 0.8B

A compact vision-language model for grounding desktop UI instructions

Version 1 · Experimental fine-tune · Bounding-box and click-point prediction

A first step toward giving desktop agents a lightweight visual grounding component: turn a natural-language UI instruction and a screenshot into a target location. :::

---

Overview

AgentEye Grounder 0.8B v1 is an experimental fine-tuned vision-language model designed to identify the screen region associated with a user interface instruction.

Given a screenshot and an instruction such as:

"Tap the Work favorite."

the model is trained to return a JSON response containing a bounding box and a click point:

{
  "bbox": [9, 222, 192, 259],
  "point": [101, 240]
}

The numbers above illustrate the expected output format; they are not a guaranteed prediction for every screenshot.

This model is intended as a potential visual-grounding component for desktop agents, including the Agent's EYE / Desk Agent direction. It predicts locations; it does not itself operate the mouse or keyboard.

What it does

• Instruction-conditioned grounding: use a natural-language instruction to identify a UI target in a screenshot.
• Bounding-box prediction: return [x1, y1, x2, y2] for the predicted target region.
• Click-point prediction: return [cx, cy] as a candidate point inside or near the target.
• Compact model scale: fine-tunes a small vision-language base model rather than training a large model from scratch.
• Agent integration potential: output can be consumed by a separate executor that validates coordinates before acting.

Model details

---

Field Details

---

Suggested model name AgentEye-Grounder-0.8B-v1

Model type Vision-language model (VLM)

Base model unsloth/Qwen3.5-0.8B (confirm this matches the exact checkpoint used for training)

Primary task Screenshot UI grounding

Input Screenshot + natural-language UI instruction

Output JSON with bbox and point

Training dataset Salesforce/grounding_dataset

Training stage Version 1, initial fine-tune

Training framework Unsloth + Hugging Face ecosystem

Intended use Research and experimentation with desktop agents

Before publishing: replace the Hugging Face links/placeholders below with the actual model repository URL and verify that the base-model name matches the checkpoint you trained.

Training summary

The provided training log reports the following configuration:

Setting Value

---

Dataset examples reported 67,356 Epochs 1 Total steps shown in the supplied log 3,000 Per-device batch size 2 Gradient accumulation steps 4 Data-parallel GPUs 1 Effective batch size 8 Trainable parameters 13,181,952 Total parameters 866,167,872 Trainable parameter share 1.52%

Step-count note: the project description says this is a 30,000-step version, but the supplied Unsloth log says Total steps = 3,000. These values conflict. Confirm the actual completed step count from the training run before publishing the model card. Do not label the checkpoint as trained for 30,000 steps unless the run really completed that many steps.

The trainable parameter count indicates that only a small portion of the full model was updated, consistent with parameter-efficient fine-tuning. The exact adapter configuration (for example, LoRA rank, target modules, learning rate, optimizer, scheduler, sequence length, image resolution, random seed, and hardware type) was not included in the supplied details and should be added when confirmed.

Dataset

Training dataset: Salesforce/grounding_dataset

The training data was described as containing 67,356 examples. The model's target format is a JSON object with:

• bbox: [x1, y1, x2, y2]
• point: [cx, cy]

Dataset-specific details such as the exact split, filtering rules, coordinate convention, license, and any preprocessing or augmentation should be checked against the dataset card and the actual preprocessing code before making further claims.

Please review and comply with the dataset's license and terms before redistributing model weights or using the model commercially.

Quick start

The examples below assume that you have published a compatible merged model to Hugging Face. Replace YOUR_HF_USERNAME/AgentEye-Grounder-0.8B-v1 with your actual repository ID.

1. Install dependencies

Run in a Python environment with a supported CUDA GPU:

pip install -U unsloth transformers pillow accelerate

For a fresh environment, follow the current installation instructions for Unsloth and make sure your PyTorch/CUDA versions are compatible. GPU inference is recommended; CPU inference may be significantly slower.

2. Load the model

import torch
from unsloth import FastVisionModel

MODEL_ID = "Navanjana/AgentEye-Grounder-0.8B-v1-GGUF"

model, tokenizer = FastVisionModel.from_pretrained(
    model_name=MODEL_ID,
    load_in_4bit=False,  # use True only if the published model/Unsloth setup supports it
    auto_model=True,
)

FastVisionModel.for_inference(model)

If you publish an adapter-only checkpoint rather than a merged model, load the exact compatible base model first and then load the adapter according to the Unsloth/PEFT instructions. An adapter-only repository is not interchangeable with a merged model repository.

3. Run a prediction on a screenshot

import json
import torch
from PIL import Image

# Replace with the path to your screenshot.
image = Image.open("screenshot.png").convert("RGB")
instruction = "Click the search icon."

messages = [
    {
        "role": "system",
        "content": (
            "You are a screen executor. Given a screenshot and a clear "
            "instruction, locate the target UI element and return its position. "
            'Always reply in this exact JSON format: '
            '{"bbox": [x1, y1, x2, y2], "point": [cx, cy]}'
        ),
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": instruction},
        ],
    },
]

prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
)

inputs = tokenizer(
    image,
    prompt,
    add_special_tokens=False,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
    )

# Decode only the newly generated tokens.
prompt_length = inputs["input_ids"].shape[1]
new_tokens = output[0][prompt_length:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True)

print("Model response:")
print(response)

# Optional: parse JSON if the model returned valid JSON.
try:
    result = json.loads(response)
    print("Bounding box:", result.get("bbox"))
    print("Click point:", result.get("point"))
except json.JSONDecodeError:
    print("The response was not valid JSON; inspect the raw output.")

Compatibility note: Unsloth's vision model loading and processor APIs can vary by model and package version. Test this snippet against the exact exported checkpoint and installed Unsloth version. If the model requires a model-specific processor API, use the loading/inference pattern from the original training notebook.

Kaggle: upload your own image and test

This optional example is intended for a Kaggle notebook after the model and tokenizer have been loaded. It uses a file-upload widget; widget availability can depend on the notebook environment.

from io import BytesIO
from PIL import Image
from IPython.display import display
import ipywidgets as widgets

uploader = widgets.FileUpload(accept="image/*", multiple=False)
display(uploader)

After uploading an image, run:

# Supports common ipywidgets 7 and 8 return formats.
if not uploader.value:
    raise ValueError("Upload an image first.")

if isinstance(uploader.value, dict):
    uploaded_file = next(iter(uploader.value.values()))
else:
    uploaded_file = uploader.value[0]

image_bytes = uploaded_file["content"]
image = Image.open(BytesIO(image_bytes)).convert("RGB")
display(image)

instruction = input("Enter a UI instruction: ")

Then run the inference code in the previous section using this image and instruction.

Visualize the predicted box

After the inference code has produced response, you can draw the predicted bounding box and point:

import json
import re
from PIL import ImageDraw
import matplotlib.pyplot as plt

match = re.search(r"\{[\s\S]*\}", response)
if not match:
    raise ValueError("No JSON object found in the model response.")

result = json.loads(match.group())
bbox = result.get("bbox")
point = result.get("point")

if not (isinstance(bbox, list) and len(bbox) == 4):
    raise ValueError("Expected bbox: [x1, y1, x2, y2].")

preview = image.copy()
draw = ImageDraw.Draw(preview)
draw.rectangle(bbox, outline="red", width=4)

if isinstance(point, list) and len(point) == 2:
    x, y = point
    r = 6
    draw.ellipse((x-r, y-r, x+r, y+r), fill="lime", outline="black", width=2)

plt.figure(figsize=(14, 8))
plt.imshow(preview)
plt.axis("off")
plt.show()

The visualization assumes the predicted coordinates are in the original screenshot's pixel coordinate system. Verify the training labels and preprocessing pipeline before relying on that assumption.

Publishing to Hugging Face

Use a Hugging Face access token with permission to create or write to your model repository. Do not hard-code a token into a notebook, script, or public repository.

Recommended: publish a merged model

In the training notebook, after training and before the runtime ends:

from huggingface_hub import login
import os

# Recommended: set HF_TOKEN as a Kaggle Secret or environment variable.
# Never paste a real token into a public notebook.
login(token=os.environ["HF_TOKEN"])

REPO_ID = "Navanjana/AgentEye-Grounder-0.8B-v1-GGUF"

model.push_to_hub_merged(
    REPO_ID,
    tokenizer,
    token=os.environ["HF_TOKEN"],
    # Use the save/quantization options supported by your installed Unsloth version.
)

Check Unsloth's current export documentation for the correct merged precision and supported options for this particular vision model. A merged 16-bit export is a practical choice for a broadly compatible initial release, but it can require substantial RAM, disk space, and upload time.

Alternative: publish a GGUF export

GGUF support and quantization options depend on the model architecture and Unsloth version. Use this only after verifying that the trained vision-language model's architecture and multimodal components are supported by the selected GGUF export path.

Example pattern from Unsloth (verify current compatibility first):

model.push_to_hub_gguf(
    "Navanjana/AgentEye-Grounder-0.8B-v1-GGUF",
    tokenizer,
    quantization_method="q4_k_m",
    token=os.environ["HF_TOKEN"],
)

Do not assume a GGUF export preserves all vision capabilities unless you have tested image input end to end.

Limitations and responsible use

• Experimental release: this is the first iteration, not a validated production-grade UI automation model.
• No published benchmark yet: no measured grounding accuracy, IoU, click-point accuracy, latency, or cross-application evaluation was supplied.
• Coordinate errors are possible: always validate the predicted box and point before taking an action.
• UI changes affect results: themes, scaling, resolution, localization, dialogs, and application updates may affect predictions.
• Instruction ambiguity matters: instructions should identify a specific visible target.
• JSON is not guaranteed: the model may occasionally return malformed JSON or coordinates outside the image bounds.
• Safety boundary: this model predicts a location; it should not be given unchecked authority to perform sensitive actions such as payments, account changes, deleting files, or submitting forms.
• Privacy: screenshots may contain personal or confidential information. Handle them according to the user's privacy requirements and avoid uploading them to third-party services without permission.

For an executor, validate that 0 <= x1 < x2 <= image_width and 0 <= y1 < y2 <= image_height, that the click point is in bounds, and that the target is consistent with the instruction before interacting with the operating system.

Roadmap

Planned direction for future versions:

• Train longer and tune the data pipeline based on evaluation results.
• Build a held-out benchmark for bounding-box IoU, point accuracy, JSON validity, and inference latency.
• Test across multiple applications, display resolutions, scaling factors, and UI themes.
• Improve robustness to ambiguous instructions, small icons, and visually similar controls.
• Compare base, adapter, and merged checkpoints consistently.
• Integrate predictions into Agent's EYE / Desk Agent behind a verification layer.
• Publish reproducible training configuration and evaluation results with later releases.

These are future goals, not capabilities claimed for this v1 checkpoint.

Citation

If you use this model, cite the model repository once it is published and cite the original dataset:

@misc{agenteye_grounder_v1,
  title        = {AgentEye Grounder 0.8B v1},
  author       = {Navanjana},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  note         = {Experimental vision-language model for desktop UI grounding}
}

Dataset: Salesforce/grounding_dataset

Acknowledgements

• Unsloth for fine-tuning and model-export tooling.
• Hugging Face for model and dataset hosting.
• The authors and maintainers of Salesforce/grounding_dataset.

---

AgentEye Grounder · Version 1

Small visual grounding models for more capable desktop agents.