ModernBERT field-event ranker (mixed-mode v2.1, H4-320)
Fine-tune of answerdotai/ModernBERT-base with v2.1 labels — same as v2
but with a sharpened action_causality heuristic:
- Old: tool_result is grounded if ≥2 content tokens appear in ANY later
assistant turn. Noise-dominated (
file,path,error,exitcodematch trivially). - New: require ≥2 distinctive tokens (filter out 50+ corpus-frequent noise tokens) and the assistant turn must be the NEXT one, within 2 events. Selects a proper subset of tool_results based on content grounding.
Key result: per-field improvements on sharpened fields
| Field | v2-320 | v2.1-320 | Δ |
|---|---|---|---|
| action_causality | 0.561 | 0.698 | +0.137 |
| failed_attempts | 0.705 | 0.845 | +0.140 |
| recent_error | 0.826 | 0.905 | +0.079 |
| customer_identity | 0.811 | 0.864 | +0.053 |
| next_actions | 0.755 | 0.786 | +0.031 |
| initiating_command | 0.966 | 0.968 | +0.002 |
| outcome_of_latest_attempt | 0.966 | 0.938 | -0.028 |
| attempted_actions | 0.480 | 0.473 | -0.007 |
| observed_outcomes | 0.491 | 0.486 | -0.005 |
Overall MRR stayed flat at 0.583 (aggregate), but top-1 moved from 0.375 → 0.381. action_causality's eval mass shrank (141 → 50 pairs) because the sharper heuristic only emits the field when the signal is real, which is why the big per-field MRR gain doesn't translate 1:1 into aggregate gain.
Training details
- Train rows: 1217
- Eval rows: 220
- Steps: 320 × accum=8 = 2560 forwards
- Last loss: 0.0007
- Peak VRAM: 4.08 GiB
Eval metrics
| metric | value |
|---|---|
| overall MRR | 0.583 |
| top-1 | 0.381 |
| top-2 | 0.576 |
| top-3 | 0.722 |
| top-5 | 0.904 |
| mean expected rank | 2.69 |
Per-field MRR (all fields):
| field | n | MRR | top-1 |
|---|---|---|---|
attempted_actions |
846 | 0.473 | 0.24 |
observed_outcomes |
845 | 0.486 | 0.24 |
initiating_command |
199 | 0.968 | 0.94 |
outcome_of_latest_attempt |
181 | 0.938 | 0.88 |
attempt_outcome_pairs |
53 | 0.608 | 0.34 |
action_causality |
50 | 0.698 | 0.48 |
failed_attempts |
36 | 0.845 | 0.72 |
next_actions |
32 | 0.786 | 0.59 |
recent_error |
29 | 0.905 | 0.83 |
customer_identity |
11 | 0.864 | 0.73 |
transaction_reference |
5 | 0.583 | 0.40 |
product_name |
4 | 1.000 | 1.00 |
discarded_options |
2 | 0.292 | 0.00 |
invalidation_hints |
2 | 0.625 | 0.50 |
non_promotable_context |
2 | 0.375 | 0.00 |
payment_or_warranty_detail |
2 | 0.170 | 0.00 |
assistant_claims_to_verify |
1 | 1.000 | 1.00 |
explicit_decisions |
1 | 1.000 | 1.00 |
resolved_context |
1 | 0.333 | 0.00 |
touched_files |
1 | 0.333 | 0.00 |
unsupported_hypotheses |
1 | 0.250 | 0.00 |
Lineage
| checkpoint | MRR | top-1 | note |
|---|---|---|---|
| mixed-v1 perf-H4 | 0.506 | 0.260 | role-tautological labels |
| mixed-v2 perf-H4 (160) | 0.526 | 0.296 | semantic labels, non-converged |
| mixed-v2 H4-320 | 0.583 | 0.375 | converged |
| mixed-v2.1 H4-320 (this) | 0.583 | 0.381 | sharpened causality |