claes preview

The multilingual Laya decision model (mmBERT-base encoder, 322M params) fine-tuned on syvai/danish-dynaword-laya (ctx1024 config): 337k typed questions over Danish Dynaword texts. Laya scores each answer option of a choice/noul question at a [MASK] marker and returns calibrated probabilities. It never generates text, so the output is always one of the options you supplied.

import laya
agent = laya.load("syvai/claes")
res = agent.predict(
    "Forslaget om kontingentforhøjelse blev forkastet med 41 stemmer mod 12.",
    {"udfald": {"type": "choice", "instructions": "Udfaldet af afstemningen.",
                "criteria": {"forkastet": "", "vedtaget": ""}},
     "har_sted": {"type": "noul", "instructions": "Oplyser teksten følgende: Stedet hvor mødet blev afholdt.",
                  "criteria": {"false": "nej", "true": "ja"}}})
print(res["answers"])
# {'udfald': {'type': 'choice', 'choice': 'forkastet', 'probabilities': {'forkastet': 0.64, 'vedtaget': 0.36}, ...},
#  'har_sted': {'type': 'noul', 'noul': 0.28, ...}}

Question phrasing that matches the training data works best:

• presence: Oplyser teksten følgende: <feltbeskrivelse> (noul, criteria {"false": "nej", "true": "ja"})
• boolean / enum field: the bare field description
• field on one item of a list: Om "<record>" (<nøglefelt>): <feltbeskrivelse>

Enum options can be bare keys; adding descriptions neither helps nor hurts after fine-tuning (see ablation below).

Files: model.safetensors, rl_agent_config.json (with max_len 1024, head_max_len 256 and the fitted temperatures), tokenizer/, encoder/config.json, temperature_fit.json.

---

Evaluation report

Fine-tune of the multilingual Laya checkpoint (convaiinnovations/laya, subfolder multilingual/, mmBERT-base, 322M params) on syvai/danish-dynaword-laya, config ctx1024 (states up to 1,024 tokens). The 16k default config was not trained: the only available hardware was a Mac mini M4 with 16 GB, which runs out of memory at 8k tokens (see "What didn't work").

All metrics below use the temperature-fitted config shipped in this repo.

Setup

Danish test split (ctx1024, 6,664 questions)

Length buckets: every ctx1024 test state is ≤ 1k tokens, so the 1k–8k and 8k–16k buckets are empty. The 8k–16k regime was not evaluated (the default config was never run).

Selected sources (accuracy before → after): enevaeldens 0.48 → 0.86, kb 0.39 → 0.68, wiki 0.54 → 0.86, health 0.32 → 0.81, cellar 0.43 → 0.85, ai-aktindsigt 0.49 → 0.88, municipality 0.43 → 0.84, hest 0.32 → 0.63, opensub 0.32 → 0.50.

Validation trajectory (3,000-question subset, raw logits): baseline Brier 0.830 → 0.377 (0.1 epoch) → 0.313 (25 %) → 0.284 (50 %) → 0.272 (75 %) → 0.271 (100 %). Gains flattened in the last quarter.

English regression check: LocalLLaMA/typed-decisions test (2,000 decisions)

No regression. Note the multilingual base is near chance on this benchmark zero-shot (upstream's BENCHMARKS.md says the same), so it is a weak English check; MASSIVE English is the better signal.

MASSIVE intent (20 options, 500 utterances per language)

Danish, English and German improve; Swedish accuracy is flat (−0.4 pt, within noise; the 75 % checkpoint was 2.6 pt below base) while its Brier and ECE improve.

Temperature fitting (validation, 5,537 questions)

All fitted temperatures are > 1, i.e. the raw model is overconfident, as expected from one-hot targets.

Do option descriptions help? (400 sampled test choice questions)

Enum options in the dataset are bare keys with empty descriptions. Three renderings of the same questions:

Descriptions help the untrained model a little; after fine-tuning on bare keys the differences vanish (±2 pt noise). Whether training with descriptions raises the ceiling was not tested.

What didn't work / limitations

• 16k context not trained. MPS on 16 GB fits ~4k-token sequences for training (fwd+bwd at 4k: 8.7 s, 10 GB; 8k: OOM). Throughput at 1k was ~1.1k tok/s, so the default epoch (1.49B tokens) would take > 1 month. Needs an A100/H100. Everything here is ctx1024; states longer than 1,024 tokens are silently truncated at inference.
• First run swapped. 8,192-token micro-batches hit a 19 GB footprint and throughput fell from 1,170 to 520 tok/s over 1.3 h; restarted at 4,096 tokens with periodic torch.mps.empty_cache() (13–15 GB, stable).
• record_enum is the weak kind (0.61). Records identified by weak keys (e.g. "55" (co2_quota_price_2020)) and evidence cut off by the 1k window are the visible failure modes; some gold labels are judgment calls.
• Overconfidence, corrected by temperature (noul T = 3.55). Label smoothing was implemented (--label-smoothing) but not run.
• Dataset oddity: many test-split workflow values are document IDs (e.g. 1334, AE003077) rather than source names; per-source breakdowns for those rows are meaningless.

What to try next

1. Train the default (16k) config on a GPU with flash attention; curriculum from this checkpoint.
2. Second epoch or a ctx1024 → default curriculum: validation was still improving slowly at the end of epoch 1.
3. Label smoothing 0.05–0.1 (one flag) to reduce the fitted temperatures.
4. Mix a few percent of MASSIVE/typed-decisions data into training to protect other languages (Swedish).
5. Add short enum descriptions to the training schemas and re-run the description ablation.
6. Teacher self-agreement on the test split to get a ceiling for the LLM-made labels.