laya-browser-v1

A continuation fine-tune of cklxx/laya-browser v17s that adds Turkish web coverage and real click-trace data. Same architecture (mmBERT-base 322M encoder + decision heads), same format (laya_fmt v3), same checkpoint layout — a drop-in replacement for v17s/ in the laya-browser repo.

Like all laya System-1 models, it does not generate text. It answers typed questions (choice, noul, score) about a page state and returns the answer plus calibrated probabilities. The browser agent loop, the HTTP server and the text side (TYPE_TEXT values) come from the harness, not from this model.

• Checkpoint size: 647 MB (fp16), 5 files — model.safetensors, encoder/, tokenizer/, rl_agent_config.json
• Latency (RX 7800 XT, fp16, stock forward): ~170–480 ms per decision at 37–59 options, mean ≈ 300 ms
• Training: ~5 h, 15,987 updates, 4 epochs, peak VRAM 6.1 GB

Benchmarks at a glance

Every number below was measured locally on one RX 7800 XT 16 GB, with the same harness for every model (laya-jev-benchmark for the decision suites, finetune/eval.py for the held-out browser split). Rows marked * are quoted from the vendor, not re-measured. parent = cklxx/laya-browser v17s, base = convaiinnovations/laya. Detail and per-source breakdowns follow in Results below.

† measured by the upstream cklxx/laya-browser repo on the same held-out split (quoted base-model row).

How much better, plainly.

• Versus its parent (v17s): +4.0 points of held-out operation accuracy (0.663 → 0.703), driven by NNetNav TYPE_TEXT (0.49 → 0.62, +13) and live DONE (0.41 → 0.72, +31); element top-1 is flat (0.548 → 0.539, inside noise) and SCROLL_DOWN slips on a 59-case sample. Calibration error drops 0.257 → 0.211 (−18 % relative) and probe-suite failures halve (11 → 6, back to the base model's level). The Turkish Wikipedia task goes from FAIL — the parent clicks "Oturum aç" and declares DONE on the login page — to PASS in 2 steps.
• Versus the base model: general decision accuracy stays at the same level (0.344 vs 0.361, −1.7 points) and calibration is worse than the base's (0.211 vs 0.175) but better than the parent's (0.257). The decisive difference is capability: the base model cannot do this job at all — on the same held-out split its operation accuracy is 0.54 and element top-1 0.10 (quoted from the upstream repo) against 0.703 / 0.539 here — and phishing ranking improves by +7.7 AUROC over the base (0.759 vs 0.682).
• Versus Jev 1.13.0: on general workflow decisions Jev wins by a wide margin (0.727 vs 0.344) — it is a far larger, purpose-built model, and this checkpoint does not claim to match it. What this 322M model wins is speed and ranking: 55 ms vs 710 ms p50 (~13x faster) and a better phishing AUROC (0.759 vs 0.689), while losing badly on phishing recall (0.024 vs 0.432) — never use it as a phishing threshold without fitting a bias term.

What went into it

Initialized from v17s weights, trained with laya's RLCD recipe (finetune/train_desktop.py, LAYA_FMT=v3, LAYA_HEAD=768, MAX_TARGETS=40, FINAL_P=0.2, micro-batch 4 × accum 8).

51,557 tokenized items (1,611 steps/epoch), from 28,499 cases:

Held-out eval set: 7,456 cases (runs/browser_v1/eval_cases.jsonl), never used for training; the calibration temperature (1.7046) was fitted on a separate split of it (n=6787).

Results

All numbers below were measured on one machine (RX 7800 XT 16 GB, Chrome headless via CDP, systemone_server.py with 12-option chunks) unless marked published. Nothing here is quoted from another run.

1. Browser agent suites (the author's suites, 1 run per task, same teacher for both arms)

Failures are identical in both arms (books-page2, flights). One run of wiki-search in this model's arm first failed with a Chrome IPC timeout (_IPCResponseTimeout, a harness error, not a model error); the retry passed in 6.3 s.

Caveat, stated plainly: 7 Suite-A domains (en.wikipedia, github, news.ycombinator, arxiv, quotes.toscrape, the-internet.herokuapp, google) also appear in this model's training crawl, so Suite A is not a clean holdout for it. Suite B is verified clean (held-out check: OK). On the clean suite the two models tie.

For reference, the parent's own published numbers on the same suites (3 runs per task, the author's machine) are Suite A 85% (41/48) and Suite B 100% (54/54) — our single-run arm reproduces the same pass/fail pattern (books-page2 and flights fail in both arms).

2. Turkish task (not in training data for either arm)

Goal (Turkish): "Vikipedi'de 'yapay zeka' araması yap ve yapay zekâ maddesini aç." — start page tr.wikipedia.org/wiki/Anasayfa, 12 steps max.

3. Held-out decision accuracy (1,288-case stratified sample of the 7,456-case set)

Same sample, same machine, finetune/eval.py, LAYA_HEAD=768. target top-1 is measured only on cases with a known gold element (n=1,107).

Reading it honestly: operation accuracy improves by +4.0 points overall, driven by NNetNav TYPE_TEXT (+13) and live DONE (+31); target top-1 is flat (0.548 → 0.539, within noise) and SCROLL_DOWN regressed slightly on a small sample (n=59). The Mind2Web and NNetNav buckets are clean holdouts (test splits, never trained on). The live bucket is in-domain-ish: 85 of its 106 labelled cases sit on domains that also appear in the training crawl (same sites, different goals and states; one exact URL overlap), so treat that gain as expected in-domain behaviour, not as a generalization result.

4. Decision-quality benchmarks on Jev's published tasks (local run of the laya-jev-benchmark harness)

Jev publishes accuracy/ECE numbers for its hosted model; the open harness in datasets/laya-jev-benchmark re-runs the same tasks locally. This checkpoint is a browser/UI decision head, so these two tables are mainly a "did fine-tuning damage anything?" check — reported in full, including where it loses.

typed-decisions test split (400 cases / 2,000 decisions; these are Jev's four published workflows: security incidents, agent trace observability, invoice processing, customer service):

Per workflow (accuracy / ECE): this model 0.322/0.171 · 0.410/0.106 · 0.304/0.333 · 0.340/0.240 — base 0.388/0.204 · 0.382/0.105 · 0.362/0.226 · 0.312/0.233 (agent_trace · customer_service · invoice · security).

Reading: on general workflow decisions a 322M browser head is nowhere near a 4B-class specialised model (0.34 vs 0.727), and this model does not claim to be one. Against its own parent it is level on accuracy (0.344 vs 0.343) with a clearly better ECE (0.211 vs 0.257, Brier 0.388 vs 0.472). The base model's raw ECE (0.175) is still the best of the three — the three shipped temperatures (3 values) were fitted for it, ours only refits the first.

probe suite (11 behavioural assertions: contradiction, stability, grounding, overconfidence — fewer failures is better):

The parent's failures are concentrated in contradiction pairs (P(A)+P(B) collapsing to ≈0.1) and wrong answers given at confidence 1.00. That is consistent with a shipped configuration bug, not a knowledge gap: v17s/rl_agent_config.json carries temperature = [5.0256, 1.0, 1.0], and laya clamps the first value (valid range [0.5, 5]) with RuntimeWarning: this checkpoint ships invalid temperatures ... Treat confidence from the affected entries as uncalibrated. This checkpoint carries [1.7046, 1.0, 1.0] (valid, refit), which brings the suite back to the base model's level.

PhishNChips core split (2,000 emails, 1,000 phishing / 1,000 legitimate; Jev published 0.626 acc / ECE 0.154 / AUROC 0.689 / recall 0.432):

This is the one benchmark where this checkpoint beats both its parent and the base model on ranking (AUROC 0.759 vs 0.577 / 0.682) and accuracy (0.511). It is still a chance-level classifier — recall 2.4%, i.e. it says "not phishing" to almost everything, exactly the bias documented in the harness README (the ranking is fine, the threshold is wrong; Platt scaling on 1,000 emails lifts base laya to 0.611 but cannot be added at inference by the checkpoint). Do not use this model as a phishing filter without fitting a bias term.

Reproducibility note: our base-model row (0.505 / 0.441 / 0.682) reproduces the harness author's published base-laya row (0.505 / 0.441 / 0.678) to two decimals on a different GPU, which is a useful sanity check that the harness itself is being run correctly.

5. Latency

The Pi has no GPU offload (VideoCore is not usable by torch/onnxruntime); ~6–8 ms per input token, input-bound. Use it for nightly/batch work with states ≤300 tokens.

How to use

import laya
agent = laya.load("ikaanakkok/laya-browser-v1")   # or a local checkpoint dir
state, questions = ...   # see code/sample_request.json in the laya-browser repo
answers = agent.predict(state, questions)["answers"]

HTTP (the same server the suites above were run with):

python code/apps/systemone_server.py 8791 /path/to/this/ckpt 12   # 12 = chunk width for wide choice questions
curl -s localhost:8791/health

With the jev-ultrafast agent (MIT, Browser Use): apply code/jev-ultrafast.patch, start Chrome with --remote-debugging-port=9222, then set BU_CDP_URL=http://127.0.0.1:9222 and TYPESAFE_BASE_URL=http://127.0.0.1:8791 and run the agent loop. A recorded browser step can be checked with the repo's verify.py.

Limitations

• No text generation, no vision. It reads the accessibility text of the page; elements with no accessible name cannot be targeted. Screenshots are not consumed.
• The target head degrades as the candidate list grows — measured on the parent architecture: 2 candidates 61.7%, 4 → 36.7%, 8 → 33.3%, 16 → 26.7%, 32 → 11.7%, full list (~151) ≈ 2%. Real desktop screens average ~159 candidates. Use chunked selection (systemone_server.py does this) and treat wide-list target picks as low-confidence.
• An opt-in embedding shortlist ships with laya (laya.shortlist), but on the desktop head's held-out set mean-pooled encoder embeddings rank candidates poorly (recall@20 12.5%, versus 34% for a simple lexical baseline and 52% top-1 for a perfect shortlist at 8 candidates), so chunked selection remains the recommended wide-list path.
• Trained for browser/desktop UI decisions, English + Turkish. Do not use it for authentication, payments, or irreversible actions without a human check.
• Part of the Turkish crawl labels (1,416 of 51,557 items, ≈2.7%: 843 goal/label cases + 573 second-step variants) were produced by a hosted teacher model; the 200 click traces and everything else are public data or generated from real executions.

Provenance & credits

• Base: convaiinnovations/laya (Apache-2.0) → cklxx/laya-browser v17s (Apache-2.0) → this checkpoint. This repository ships weights + this model card only (no code/): harness references below point at the upstream cklxx/laya-browser repo. The parent is Apache-2.0; this derivative is published under MIT with the upstream notices kept.
• Suites A/B and the decision benchmarks: cklxx/laya-browser (code/apps/browser_suite.py, browser_suite_b.py, datasets/laya-jev-benchmark), Apache-2.0.
• Reference numbers for the decision benchmarks are TypeSafe's published Jev results (quoted as such).
• Agent harness: browser-use/jev-ultrafast, MIT.
• Training data licenses: Mind2Web and NNetNav are used for training only, under their own terms.

Türkçe özet

cklxx/laya-browser v17s'in devamı niteliğinde bir ince ayar: Türkçe site kapsaması ve gerçek tıklama izleriyle eğitildi. Aynı mimari, aynı format — v17s'in yerine doğrudan kullanılabilir. Metin üretmez; sayfa durumu hakkında tipli soruları yanıtlar ve kalibre edilmiş olasılık döndürür. İngilizce suite'lerde v17s ile başa baş (Suite B 18/18, Suite A 14/16) ve biraz daha hızlı; Türkçe görevde v17s başarısız olurken bu model başarılı (tr.wikipedia "yapay zeka" görevi, 2 adım, 8,0 s).