laya-typed-decisions-ce-1024

A ModernBERT-large decision model fine-tuned on LocalLLaMA/typed-decisions with plain cross-entropy against the teacher's soft targets, at the documented 1024-token context / 256-token option budget.

It is the best-performing checkpoint from the Sev study (Dissecting RLCD), where I took apart the training method behind TypeSafe's Jev and its open reproduction, Laya. The short version of that study: Laya's RL term is a noise-smoothed cross-entropy gradient, and on this benchmark plain CE matches or beats it on every proper score.

Test-set results (2,000 decisions, 400 cases):

For reference, the laya-typed-decisions checkpoint reports 0.766 accuracy and 0.062 Brier; TypeSafe's published Jev 1.13.0 figure is 0.727.

Model description

A bidirectional encoder plus a small transformer head that reads one logit per masked option marker:

[CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP]
                                                        |
                                  ModernBERT-large encoder (bidirectional)
                                                        |
                                  h += type_emb(qtype)
                                                        |
                                  2 x TransformerEncoderLayer (pre-norm)
                                                        |
                                  gather h at [MASK] positions -> scorer MLP -> 1 logit/option

One input row per typed question. Question types are choice (pick one of K named options), score (ordinal), and noul (binary yes/no). Option count is per-row, not fixed.

• Encoder: answerdotai/ModernBERT-large (395M)
• Head: 2 pre-norm transformer layers, dropout 0.1
• Scorer: LayerNorm, Linear, GELU, Linear(->1)
• Total: ~421M parameters
• Sequence budget: 1024 tokens total, 256 for the question + option block

The output is a probability distribution over the row's options. For the score type, the expected level is sum i * p_i.

Intended use

Structured, closed-set probabilistic decisions over short English text: routing, triage, classification with calibrated confidence, and ordinal rating tasks where you control the option set.

Out of scope: open-ended generation (this model does not generate text), option sets larger than ~255, languages other than English (the base is English; a multilingual Laya variant exists), and any use where the option wording and the state distribution differ sharply from the training workflows.

How to use

This is a custom architecture, not a transformers AutoModel. Load it with the code from the reproduction repo:

git clone https://github.com/LakoreAI/sev
cd rlcd-reverse-engineering && uv sync

from pathlib import Path
import torch
from huggingface_hub import snapshot_download

import sys
sys.path.insert(0, "rlcd-reverse-engineering")  # or pip install -e it

from src.pipelines.infer import load_model, infer

d = snapshot_download("minhleduc/laya-typed-decisions-ce-1024")
model, cfg, tokenizer = load_model(Path(d) / "model.safetensors", torch.device("cuda"))

result = infer(
    Path(d) / "model.safetensors",
    state='{"task": "triage", "ticket": "customer cannot log in after password reset"}',
    question='{"type": "choice", "instructions": "Which queue should this go to?",'
             ' "criteria": {"billing": "payment or invoice", "auth": "login or account access",'
             ' "network": "connectivity"}}',
)
print(result["predicted_key"], result["probabilities"])

load_model reads model_config.json to rebuild the architecture and temperatures.json to apply the fitted per-(type, K-bucket) temperature at inference, so the reported probabilities are already calibration-scaled.

Training data

LocalLLaMA/typed-decisions (Apache-2.0): 1,200 train cases and 400 test cases across four workflows (agent-trace observability, customer service, invoice processing, security incidents), flattened to ~5,400 training decisions. Targets are the dataset's soft teacher distributions, not the hard labels.

Training procedure

• Initialised from convaiinnovations/laya.
• Loss: soft cross-entropy against the teacher distributions only (no RL term).
• 4 epochs, effective batch 64 (micro-batch 8 x 4 accumulation on a single 24 GB GPU), AdamW, lr 2.5e-5 encoder / 1e-4 head, weight decay 0.01, cosine decay to 1e-6, gradient clipping 1.0, bf16.
• No early stopping; the final-epoch weights are evaluated.
• Post-hoc temperature fitted per (question type, option-count bucket) by NLL on a held-out 10% slice of the training cases.

Evaluation

Full test-set breakdown by question type (2,000 decisions):

A calibration caveat that matters here. On this benchmark the gold hard label is the teacher's argmax on 98.4% of rows, so hard-label ECE is not a valid calibration measure: a model that reproduces the teacher perfectly still scores 0.326 hard-label ECE, and over-sharpening lowers it. The raw hard-label ECE of this checkpoint is 0.144 (0.162 after temperature). Judge calibration against the soft targets instead, with Brier and NLL above. The fitted temperatures (all > 1) are the model's own signal that the raw logits were slightly over-sharp before scaling.

Relation to RLCD

This checkpoint is deliberately RL-free. In the accompanying study, adding Laya's RL term (a score-function estimate of a noise-smoothed proper score) did not improve accuracy and made the fitted temperature rise with the training noise scale. CE-only was at least as good on every proper score. See the linked repo for the full ablation, the estimator derivation, and the per-run result files.

Limitations

• Single benchmark; the four workflows are synthetic and teacher-generated.
• English only.
• Option sets above ~20 options degrade, because options share a fixed token budget.
• Accuracy differences of ~1 point are within seed noise on a dataset this size; three seeds gave 0.782 +/- 0.004 for the 512-token variant.
• The base checkpoint was already fine-tuned by the Laya authors; this model is a further fine-tune on the same benchmark's train split.

Links

• Code, configs, and per-run results: LakoreAI/sev
• All ablation checkpoints and metrics: minhleduc/rlcd-e2-checkpoints
• Base model: convaiinnovations/laya
• Dataset: LocalLLaMA/typed-decisions

Citation

@misc{leduc2026dissectingrlcd,
  title  = {Dissecting RLCD: What Reinforcement Learning Does (and Doesn't) Do for Calibrated Typed Decisions},
  author = {Le Duc Minh},
  year   = {2026},
  url    = {https://github.com/LakoreAI/sev}
}