Gemma 4 12B — Web Browser Agent (Run 14)

A LoRA adapter fine-tuned on top of Google's QAT Gemma 4 12B for web navigation action prediction. Given a screenshot of a browser and a natural-language task, the model predicts the next action to take (click, type, scroll, navigate, etc.) with chain-of-thought reasoning.

Model Details

Training Data

Fine-tuned on allenai/MolmoWeb-HumanTrajs — indices 0–1999 (~2 000 trajectories, ~34 K steps). Indices 2000–2019 were held out for evaluation (never seen during training).

Each training step: screenshot → action prediction with teacher-forced prior-action history (last 10 steps).

Action Space

mouse_click(x, y, button='left')       — click at pixel (x, y)
keyboard_type(text)                    — type into focused field
scroll(delta_x, delta_y)              — scroll by pixel delta
scroll_at(x, y, delta_x, delta_y)     — scroll at position (x, y)
goto(url)                              — navigate to URL
go_back()                              — browser back
keyboard_press(key)                    — press a key (Enter, Tab, Escape…)
new_tab()                              — open new tab
tab_focus(index)                       — switch to tab by index
send_msg_to_user(message)             — report result to user (terminal action)

Coordinates are pixel values relative to a 1920×1200 viewport.

Evaluation Results

Evaluated on 20 held-out trajectories (314 steps) from MolmoWeb-HumanTrajs (indices 2000–2019) using teacher-forced ground-truth history (mirrors training format).

Action match +27 pp is the headline result — the model learned the action vocabulary well. Coordinate regression (−29 pp) is the main open issue: the fine-tuned model produces fewer click/scroll steps and more out-of-range coordinates; this is the next thing to fix.

Full per-step results are in eval_run14_results.json in this repository.

How to Use

import torch
from peft import PeftModel
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

BASE = "google/gemma-4-12B-it-qat-q4_0-unquantized"
ADAPTER = "medelharchaoui/gemma4-12b-browser-agent-run14"

processor = AutoProcessor.from_pretrained(BASE)
base = AutoModelForImageTextToText.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

# screenshot: PIL.Image of the current browser state (1920×1200 recommended)
screenshot = Image.open("screenshot.png").convert("RGB")
task = "Go to https://www.google.com/ and search for 'latest AI news'"

SYSTEM_PROMPT = (
    "You are a web navigation agent. You see a screenshot of the current browser state "
    "and a task to complete. Think through what to do next, then output exactly one action.\n\n"
    "Available actions (coordinates are pixel values):\n"
    "  mouse_click(x, y, button='left')\n"
    "  keyboard_type(text)\n"
    "  scroll(delta_x, delta_y)\n"
    "  scroll_at(x, y, delta_x, delta_y)\n"
    "  goto(url)\n"
    "  go_back()\n"
    "  keyboard_press(key)\n"
    "  new_tab()\n"
    "  tab_focus(index)\n"
    "  send_msg_to_user(message)\n\n"
    "When provided, 'Previous actions' lists what you have already done in this task."
)

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
    {"role": "user", "content": [
        {"type": "image", "image": screenshot},
        {"type": "text",  "text": f"Task: {task}"},
    ]},
]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(
    text=prompt, images=[[screenshot]],
    return_tensors="pt", truncation=True, max_length=3072,
).to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=128, do_sample=False)

response = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
# Example output:
# I can see the browser is open. I will navigate to Google.com to start the task.
# goto(url='https://www.google.com/')

Limitations

• Coordinate precision: the model predicts pixel coords for clicks/scrolls but with non-trivial error (~300 px avg L2 on held-out data). Suitable for coarse navigation; precise element targeting needs improvement.
• Teacher-forced history at eval: results above assume ground-truth prior actions are injected in the prompt. In a real rollout (auto-regressive history), error accumulates over trajectory steps.
• Domain coverage: trained on MolmoWeb-HumanTrajs which covers a mix of popular websites. Performance may degrade on niche or highly dynamic UIs.
• Language: English-only tasks tested.

Reproduce / Continue Training

Training code and eval scripts: OptimiAI/Gemma-browser (private repo).

To resume from this checkpoint:

# in src/train.py, set:
ADAPTER_PATH = "medelharchaoui/gemma4-12b-browser-agent-run14"
# and pass resume_from_checkpoint=ADAPTER_PATH to SFTTrainer