laya-kannaka-evidence-gate

A Laya checkpoint (ModernBERT-large, 421M, non-autoregressive typed decisions) fine-tuned on one decision that a memory substrate makes at recall time:

noul — Does the excerpt contain information needed to answer the question?

State: {"question": "...", "excerpt": "role: turn text"}. Answer: P(yes) in the noul field, calibrated.

It is the first trained reflex for Kannaka, a wave-interference memory for persistent agents, from experiment E-L1b in kannaka-bench (experiments/laya_reflex/).

Results (held-out, pre-registered rule)

Test set: the 450 (question, candidate) decisions the Kannaka medium actually produced for the 30 standard LongMemEval-S questions at k=15, none of which appeared in training.

Pre-registered rule (AUROC ≥ 0.85 and Brier ≤ 0.15): passed. At P(yes) ≥ 0.5 it keeps 41 of 43 evidence turns while flagging 69 of 450 candidates — as a gate it hands a reader ~2.3 rows per question instead of 15 and keeps 95% of the evidence. Whether that improves answers is the next experiment (E-L1c), not a claim here.

Training

• Data: 4,180 (question, turn) rows built by build_e_l1b_dataset.py from LongMemEval-S — every turn with has_answer from the 470 questions outside the standard 30 (836 positives) plus 4 sampled negatives per positive, drawn first from the same session as a positive (3,320 hard negatives of 3,344). Hard 0/1 targets.
• Recipe: upstream's RLCD notebook (zero-mean noisy-logit groups scored by proper scoring rules + soft cross-entropy), reduced to a single GPU (train_e_l1b.py): 4 epochs, micro-batch 8, grad accumulation 8, encoder LR 2.5e-5, head LR 1e-4, cosine schedule, fp16, gradient checkpointing. Post-training temperature fit: choice 1.637, score 1.251, noul 4.976.
• Compute: one RTX 4090 on qBraid, 390 s, ~5 steps/s. About 26 qBraid credits (≈ $0.26) for the instance including both evaluations.

Limits

• n = 43 positives in the held-out set; the AUROC's standard error is roughly ±0.03.
• Negatives were sampled per question from the same haystack; cross-question distractors as a production store would present them were not in training.
• English only (the base checkpoint's tokenizer/encoder). Context 1024 tokens; excerpts are cut at 2,000 characters.
• The choice and score heads were not trained on anything new and should be treated as the base model's.

Use

import laya
agent = laya.Agent("flaukowski/laya-kannaka-evidence-gate")   # or a local directory
res = agent.predict(
    {"question": "What did I say my sister does?", "excerpt": "user: my sister is a nurse in Leeds"},
    {"evidence": {"type": "noul", "instructions": "Does the excerpt contain information needed to answer the question?"}},
)
print(res["answers"]["evidence"]["noul"])   # P(yes)

Provenance

Base model Apache-2.0 (convaiinnovations). Training data derived from LongMemEval (Wu et al.), used under its license for research. Produced 2026-09-22 by Kannaka Labs; wins and losses of the surrounding experiments are in kannaka-bench RESULTS.md.