Laya-421M-Claviger-v0
A claviger is a keeper of keys. This one is a fine-tune of Laya that reads one Discord message and decides whether an AI agent in the channel should reply, and which one. It was built for citadel, a personal multi-agent setup, as an alternative to replying only on a mention or a reply.
It was trained and tested on synthetic conversations only. It has never run on real messages. Treat its numbers as optimistic.
Model Details
Model Description
- Developed by: finis-est
- Model type: decision model. It picks one option from a list and returns probabilities. It doesn't generate text.
- Architecture: ModernBERT-large encoder with Laya's decision head, 421M parameters
- Language: English
- License: Apache 2.0
- Fine-tuned from:
convaiinnovations/laya, revision7b928d828b7b0e022f929d9bd2e44165aa270148
Model Sources
- Base model: convaiinnovations/laya
- Encoder: answerdotai/ModernBERT-large
- Training data: finis-est/claviger-triage-synthetic. Training code is not public.
Uses
Direct Use
Triage for chat agents that share a channel with people. Give it the channel, the message and recent context, plus the agents present as options. It returns none or an agent.
Downstream Use
Behind a stricter rule, with the model allowed to veto or to answer only at high confidence. See Evaluation. On its own it answers too often.
Out-of-Scope Use
- Moderation, or any decision about a person
- Languages other than English
- Automatic replies with no fallback or human oversight
- Agent rosters other than the one it was trained on, without your own evaluation
Bias, Risks, and Limitations
- Synthetic data. Four Claude agents wrote every message and Claude models did the labelling. Real chat is messier, and the writers share one idea of what Discord sounds like.
- Not real traffic. The test mix is built around hard cases. Real channels are mostly banter, so production precision will be lower at the same false-answer rate.
- Small test set. 465 items from 60 conversations, so confidence intervals are wide.
- Over-confident. On the 130-item dev set, mean confidence was 0.865 against 0.808 accuracy. The fitted temperature hit its clamp at 5.0, on a 53-item calibration slice. Answers at confidence 0.9 or above were 86.6% right, not 90%.
- Open questions. It answers few unaddressed help questions, by design. This version treats them as
none. - Tied to its roster. Options are passed at run time, so other agent names work technically. A quick check on four invented messages with invented agents got 2 right. It handled a direct ask and banter, answered a complaint about an agent, and missed an ask by role.
- Frozen layers. 12 of 28 encoder layers were frozen to fit 6 GB of VRAM. A full fine-tune might do better.
Recommendations
Run it in shadow beside your existing rule first, on your own messages. Recalibrate on a held-out set of a few hundred items before you trust its confidence. Prefer missing a message to butting in.
How to Get Started with the Model
pip install laya==0.4.9
import laya
agent = laya.load("finis-est/Laya-421M-Claviger-v0", device="cpu")
question = {"who": {
"type": "choice",
"instructions": "Which AI agent in this Discord channel should reply to the message? "
"Pick none unless the message is meant for an agent: an agent should not "
"butt in on members talking to each other or talking about an agent.",
"criteria": {
"none": "no agent: members talking among themselves, to another member, or about an agent",
"mira": "Mira: the community's host: answers questions, shares guides. Answers in chat.",
"bolt": "Bolt: the community's developer: code and the website. Speaks when named.",
},
}}
state = "Channel: #general\nMessage from robin: Mira, where's the setup guide again?"
answer = agent.predict(state, question)["answers"]["who"]
print(answer["choice"], answer["probabilities"])
# mira {'none': 0.013, 'mira': 0.979, 'bolt': 0.009}
The state is plain text: the channel, the message, then mention, reply and recent context lines.
Training Details
Training Data
150 invented conversations with 1,127 human messages, split by conversation and frozen by hash. Handles, messages and agent replies are all made up. No real member data was used.
| Split | Items | Source |
|---|---|---|
| train | 532 | three writers, 53 held back for calibration |
| dev | 130 | three writers |
| test-a | 218 | three writers |
| test-adv | 247 | a fourth writer, hard cases only |
Labels record whether an agent should answer, which agent, and the kind of case. A second agent on a different model labelled blind. They agreed on 1,122 of 1,127 items, and that figure is inflated because labellers could see whether an agent replied next. A stricter check on 160 items, showing prior context only, agreed on 156 (97.5%).
The label policy is that an agent butting in is worse than an agent missing a message. When in doubt, the answer is none.
Training Procedure
Laya's own fine-tune loop (laya.train.finetune, laya 0.4.9), with option order shuffled each epoch so the model can't learn that the first option wins.
Training Hyperparameters
| Setting | Value |
|---|---|
| Epochs | 5 |
| Micro-batch, gradient accumulation | 8, 2 |
| Encoder learning rate | 2.5e-05 |
| Head learning rate | 1e-04 |
| Minimum learning rate | 1e-06 |
| Weight decay | 0.01 |
| Gradient clip | 1.0 |
| Loss | rlcd |
| Frozen | embeddings and the first 12 of 28 encoder layers |
| Seed | 0 |
Speeds, Sizes, Times
- Training took about 10 minutes on an RTX 3060, capped at 6 GB, with a 4.8 GB peak.
- The checkpoint is 803 MB.
- On CPU it uses about 4.7 CPU-seconds per message and 2.1 GB of memory.
Evaluation
Testing Data, Factors & Metrics
Testing Data
465 held-out items from 60 conversations: test-a (218) and test-adv (247). 172 should be answered and 293 shouldn't.
Factors
Results are reported overall and by split. The adversarial split is much harder.
Metrics
Precision, recall and F1 are on whether to answer. Correct agent is the share of should-answer items routed to the right agent. A false answer is an answer on an item that shouldn't get one, reported for banter and for all negatives.
Results
These results are self-reported on a private, self-made test set. No public benchmark exists for this task, and the model hasn't been run against one.
| Contender | Precision | Recall | F1 | Correct agent | False answers, banter | False answers, all negatives |
|---|---|---|---|---|---|---|
| Reply on mention or reply only (rule) | 66.7 | 24.4 | 35.7 | 24.4 | 1.7 | 7.2 |
| Laya base, zero-shot | 39.1 | 72.7 | 50.8 | 58.7 | 64.1 | 66.6 |
| This model | 72.5 | 70.3 | 71.4 | 69.2 | 6.0 | 15.7 |
| Rule plus this model, strict | 83.0 | 51.2 | 63.3 | 51.2 | 1.7 | 6.1 |
| Claude Haiku 5.5 | 68.4 | 91.9 | 78.4 | 89.5 | 12.8 | 24.9 |
The strict setup sends a mention or reply to the rule unless the model gives none 0.1 or more, and otherwise answers only at 0.95 or more. It costs no more false answers than the rule alone and doubles recall.
| This model, by split | Precision | Recall | F1 | Correct agent | False answers, banter | False answers, all negatives |
|---|---|---|---|---|---|---|
| test-a | 81.5 | 88.2 | 84.7 | 87.1 | 4.8 | 12.8 |
| test-adv | 61.3 | 52.9 | 56.8 | 51.7 | 7.4 | 18.1 |
On the 130-item dev set, accuracy went from 0.50 before fine-tuning to 0.81 after.
Summary
Fine-tuning fixed the base model's habit of answering banter, 64.1% down to 6.0%. Claude Haiku 5.5 finds more real asks but answers more than twice as much banter, and messages leave your machine. The strongest result here is the strict combination, in shadow, as a step up from a mention-only rule.
Environmental Impact
Not measured. Training ran about 10 minutes on one consumer GPU.
Technical Specifications
Model Architecture and Objective
ModernBERT-large encoder with Laya's two-layer decision head. Maximum length 512 tokens, 192 for the head. It scores each option against the state and returns probabilities.
Software
laya 0.4.9, torch 2.9.1, transformers below 5, Python 3.12.
Citation
No paper accompanies this model. ModernBERT asks to be cited as arXiv 2412.13663.
Model Card Authors
Jonas, with finis-est.