laya-guard-multilingual
A fine-tune of Laya's multilingual checkpoint (the multilingual
folder; ConvAI Innovations, Apache-2.0) for laya-guard, a local guard that judges each shell command or
tool call a coding agent wants to run as allow, review (ask the user) or block.
It is trained for one question, the one laya-guard asks (action: allow / review / block, with laya-guard's
instructions and criteria). Other questions get the base model's behaviour at best.
Use
laya-guard loads it by default from its next release. To use it directly with laya (0.4.3):
from laya import Router
router = Router(device="cpu", models={"multilingual": "Ayaannnnnnn/laya-guard-multilingual"})
Training
- Data: about 1,300 rows. Shell commands from the author's own coding-agent sessions in personal projects only (work sessions excluded), with home paths, tokens, emails and long opaque strings masked, labeled against a written guide; plus written examples (destructive and exfiltrating commands, Windows commands, secrets printed or sent, harm hidden behind a comment, tools told to run something else, Korean and English requests). Anything that copied or closely resembled (similarity ≥ 0.85) a case in laya-guard's benchmark, eval sets or test fixtures was dropped. Review and block rows were repeated twice in the training split. The data is not published.
- Run:
laya-train(laya 0.4.3) on a Colab T4, 4 epochs, 68 optimizer updates, seed 0. Tools: finetune/.
Evaluation
On sets never used for training (laya-guard's benchmark and eval sets), compared with the base model, through
laya-guard's own judging (ask=True, as for Claude Code):
| base | this model | |
|---|---|---|
| model only: attacks held (43) | 32 | 39 |
| model only: everyday commands allowed (50) | 25 | 44 |
| model only: eval shell set (71) | 42 | 61 |
| model only: Korean set (30) | 27 | 27 |
| laya-guard end to end: attacks held (43) | 41 | 42 |
| laya-guard end to end: everyday allowed (50) | 46 | 48 |
| laya-guard end to end: eval shell set (71) | 69 | 70 |
| laya-guard end to end: Korean set (30) | 30 | 29 |
Of the 8 end-to-end verdicts that changed, 6 got better and 2 got worse: chmod +x scripts/build.sh and a Korean
request to summarize an issue are now asked about instead of allowed. The pass criteria set before training asked for
no end-to-end regression, so this model misses one of them; it was adopted anyway because nothing that should be
held got through and far fewer ordinary commands are asked about.
Held-out slice of the training data (162 rows, 76% allow): accuracy 0.611 → 0.852.
Limitations
- The benchmark and eval sets were written by laya-guard's author, and they were looked at between training runs, so they are less independent than a fresh test set.
- Trained on one person's commands and on written examples; other workflows may look different.
- A model alone shouldn't have the last word: laya-guard runs its rules first and turns a model-only block into a question where the agent can ask.
License
Apache-2.0, as the base model. Base model © ConvAI Innovations.