Laya Fine-Tuned for Browser Automation (Mind2Web)
This is a fine-tuned version of the Laya decision model, specialized for browser automation tasks. It predicts the next browser action (operation and target element) from a web page's DOM structure and a user goal.
Model Details
- Developed by: Shaun Andrade (@ShaunSpark)
- Base model: convaiinnovations/laya (421M parameters)
- Training data: Mind2Web (671 usable training examples)
- Validation accuracy: 74.3% on a held-out set of 68 examples
Training Procedure
- Hardware: Google Colab, single NVIDIA T4 GPU (16 GB VRAM)
- Optimizer: 8-bit AdamW (bitsandbytes), learning rate 1e-5
- Effective batch size: 8 (micro-batch 1, gradient accumulation 8)
- Precision: FP16 mixed precision (torch.cuda.amp)
- Memory optimizations: Gradient checkpointing, expandable_segments
- Epochs: 2 (best checkpoint; further training overfits)
Intended Use
Decision engine for a browser automation agent. Given a goal and a list of interactive DOM elements, the model answers two questions in one forward pass:
- Operation: CLICK, TYPE, or SELECT
- Target: the correct element index from the candidate list
Limitations
- Trained on a single dataset (Mind2Web); may not generalize to all websites.
- Overfits beyond 2 epochs on 671 examples.
- Requires the base Laya tokenizer and architecture.
How to Use
import torch
import laya
agent = laya.load("convaiinnovations/laya")
model = agent.model
model.load_state_dict(torch.load("pytorch_model.bin", map_location="cpu"))
model.eval()