A claviger is a keeper of keys. This one is a fine-tune of Laya that reads one Discord message and decides whether an AI agent in the channel should reply, and which one. It was built for citadel, a personal multi-agent setup, as an alternative to replying only on a mention or a reply.
It was trained and tested on synthetic conversations only. It has never run on real messages. Treat its numbers as optimistic.
convaiinnovations/laya, revision 7b928d828b7b0e022f929d9bd2e44165aa270148Triage for chat agents that share a channel with people. Give it the channel, the message and recent context, plus the agents present as options. It returns none or an agent.
Behind a stricter rule, with the model allowed to veto or to answer only at high confidence. See Evaluation. On its own it answers too often.
none.Run it in shadow beside your existing rule first, on your own messages. Recalibrate on a held-out set of a few hundred items before you trust its confidence. Prefer missing a message to butting in.
pip install laya==0.4.9
import laya
agent = laya.load("finis-est/Laya-421M-Claviger-v0", device="cpu")
question = {"who": {
"type": "choice",
"instructions": "Which AI agent in this Discord channel should reply to the message? "
"Pick none unless the message is meant for an agent: an agent should not "
"butt in on members talking to each other or talking about an agent.",
"criteria": {
"none": "no agent: members talking among themselves, to another member, or about an agent",
"mira": "Mira: the community's host: answers questions, shares guides. Answers in chat.",
"bolt": "Bolt: the community's developer: code and the website. Speaks when named.",
},
}}
state = "Channel: #general\nMessage from robin: Mira, where's the setup guide again?"
answer = agent.predict(state, question)["answers"]["who"]
print(answer["choice"], answer["probabilities"])
# mira {'none': 0.013, 'mira': 0.979, 'bolt': 0.009}
The state is plain text: the channel, the message, then mention, reply and recent context lines.
150 invented conversations with 1,127 human messages, split by conversation and frozen by hash. Handles, messages and agent replies are all made up. No real member data was used.
| Split | Items | Source |
|---|---|---|
| train | 532 | three writers, 53 held back for calibration |
| dev | 130 | three writers |
| test-a | 218 | three writers |
| test-adv | 247 | a fourth writer, hard cases only |
Labels record whether an agent should answer, which agent, and the kind of case. A second agent on a different model labelled blind. They agreed on 1,122 of 1,127 items, and that figure is inflated because labellers could see whether an agent replied next. A stricter check on 160 items, showing prior context only, agreed on 156 (97.5%).
The label policy is that an agent butting in is worse than an agent missing a message. When in doubt, the answer is none.
Laya's own fine-tune loop (laya.train.finetune, laya 0.4.9), with option order shuffled each epoch so the model can't learn that the first option wins.
| Setting | Value |
|---|---|
| Epochs | 5 |
| Micro-batch, gradient accumulation | 8, 2 |
| Encoder learning rate | 2.5e-05 |
| Head learning rate | 1e-04 |
| Minimum learning rate | 1e-06 |
| Weight decay | 0.01 |
| Gradient clip | 1.0 |
| Loss | rlcd |
| Frozen | embeddings and the first 12 of 28 encoder layers |
| Seed | 0 |
465 held-out items from 60 conversations: test-a (218) and test-adv (247). 172 should be answered and 293 shouldn't.
Results are reported overall and by split. The adversarial split is much harder.
Precision, recall and F1 are on whether to answer. Correct agent is the share of should-answer items routed to the right agent. A false answer is an answer on an item that shouldn't get one, reported for banter and for all negatives.
These results are self-reported on a private, self-made test set. No public benchmark exists for this task, and the model hasn't been run against one.
| Contender | Precision | Recall | F1 | Correct agent | False answers, banter | False answers, all negatives |
|---|---|---|---|---|---|---|
| Reply on mention or reply only (rule) | 66.7 | 24.4 | 35.7 | 24.4 | 1.7 | 7.2 |
| Laya base, zero-shot | 39.1 | 72.7 | 50.8 | 58.7 | 64.1 | 66.6 |
| This model | 72.5 | 70.3 | 71.4 | 69.2 | 6.0 | 15.7 |
| Rule plus this model, strict | 83.0 | 51.2 | 63.3 | 51.2 | 1.7 | 6.1 |
| Claude Haiku 5.5 | 68.4 | 91.9 | 78.4 | 89.5 | 12.8 | 24.9 |
The strict setup sends a mention or reply to the rule unless the model gives none 0.1 or more, and otherwise answers only at 0.95 or more. It costs no more false answers than the rule alone and doubles recall.
| This model, by split | Precision | Recall | F1 | Correct agent | False answers, banter | False answers, all negatives |
|---|---|---|---|---|---|---|
| test-a | 81.5 | 88.2 | 84.7 | 87.1 | 4.8 | 12.8 |
| test-adv | 61.3 | 52.9 | 56.8 | 51.7 | 7.4 | 18.1 |
On the 130-item dev set, accuracy went from 0.50 before fine-tuning to 0.81 after.
Fine-tuning fixed the base model's habit of answering banter, 64.1% down to 6.0%. Claude Haiku 5.5 finds more real asks but answers more than twice as much banter, and messages leave your machine. The strongest result here is the strict combination, in shadow, as a step up from a mention-only rule.
Not measured. Training ran about 10 minutes on one consumer GPU.
ModernBERT-large encoder with Laya's two-layer decision head. Maximum length 512 tokens, 192 for the head. It scores each option against the state and returns probabilities.
laya 0.4.9, torch 2.9.1, transformers below 5, Python 3.12.
No paper accompanies this model. ModernBERT asks to be cited as arXiv 2412.13663.
Jonas, with finis-est.
convaiinnovations/laya