episodic-ingestion-compiler / ModernBERT field-event ranker (perf-H4, mixed-mode)
Fine-tune of answerdotai/ModernBERT-base (149 M params, 1 regression output)
for the grouped multi-positive softmax ranker task in the episodic-ingestion-compiler
training pipeline: given a requested extraction field and a set of candidate
source events in a trace, rank each event by how strongly it supports the field.
Training details
- Train set: 1201 rows across 4 tagged modes:
- 109 conversational (original hand-authored)
- 313 autonomous agent loops (generated via local GLM harness)
- 892 prompt→action (mined from local codex/claude session jsonl)
- 99 customer-support (from
jonathansuru/customer_service_information_extraction)
- Eval set: 220 rows of held-out mixed-mode data (session-hash disjoint from train)
- Steps: 160 optimizer steps (
--accum-steps 8→ 1280 forwards) - Optimizer: AdamW, lr 5.7e-5 (= 2e-5 × sqrt(8) for effective batch size 8), warmup 30
- Precision: bf16 autocast (no GradScaler)
- Memory optimizations: gradient checkpointing (
use_reentrant=False) - Final train loss: 0.0000
- Peak VRAM: 3.858 GiB (was 12.57 without perf stack)
- Hardware: single RTX 5090 Laptop (Blackwell SM 12.0, 24 GiB), CUDA 13.2, PyTorch 2.11
Eval metrics (220-row mixed eval, 1792 source pairs)
| metric | value |
|---|---|
| top-1 recall | 0.257 |
| top-2 recall | 0.491 |
| top-3 recall | 0.677 |
| top-5 recall | 0.901 |
| MRR | 0.504 |
| mean expected rank | 2.92 |
Per-field MRR (top-n fields in the eval):
| field | n | MRR | top-1 |
|---|---|---|---|
attempted_actions |
846 | 0.491 | 0.24 |
observed_outcomes |
845 | 0.491 | 0.24 |
failed_attempts |
36 | 0.866 | 0.75 |
next_actions |
32 | 0.701 | 0.47 |
customer_identity |
11 | 0.689 | 0.55 |
transaction_reference |
5 | 0.567 | 0.20 |
product_name |
4 | 0.625 | 0.25 |
discarded_options |
2 | 0.292 | 0.00 |
invalidation_hints |
2 | 0.625 | 0.50 |
non_promotable_context |
2 | 0.625 | 0.50 |
payment_or_warranty_detail |
2 | 0.312 | 0.00 |
assistant_claims_to_verify |
1 | 0.200 | 0.00 |
explicit_decisions |
1 | 0.500 | 0.00 |
resolved_context |
1 | 0.333 | 0.00 |
touched_files |
1 | 1.000 | 1.00 |
unsupported_hypotheses |
1 | 0.250 | 0.00 |
Lineage vs earlier checkpoints
| checkpoint | train rows | MRR | top-1 | notes |
|---|---|---|---|---|
| V2 (commit 87cb089) | 109 conversational | 0.678 | 0.440 | Evaluated on 8-row conversational adversarial eval — different and smaller test set |
| Stage-4 mixed-mode | 1201 mixed | 0.323 | 0.130 | Same 220-row eval as this checkpoint |
| perf-H4 (this) | 1201 mixed | 0.506 | 0.260 | bf16 + gradient checkpointing + grad accumulation + sqrt LR |
Intended use
This is the ranker head of a multi-stage episodic-ingestion compiler — NOT a
general-purpose classifier. Input is a JSON-serialized (requested field, candidate event) record from the compiler's training pipeline. Output is a
scalar logit that, after grouped softmax over siblings in the same trace,
estimates the probability that the candidate event supports the requested
field.
Limitations
- Current eval shows
attempted_actionsandobserved_outcomes(the two largest field slices, 846 + 845 of 1792 eval pairs) stuck at MRR ≈ 0.49. This is the tool_call-vs-tool_result confusion inherent to per-candidate context-free scoring. Addressing it is planned future work (marker-pool architecture at mixed-mode scale). - Training loss of 0.0001 indicates the model has converged on train. Longer training at this config is unlikely to improve eval.
- Not intended for deployment as a standalone extractor; it's one head in a larger ingestion pipeline.