OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

Reference implementation for the paper:

OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, Nikhil Mungel Cribl AI Research Lab arXiv:2610.12375

Overview

OnTrack is a streaming monitoring mechanism that compares an LLM agent's execution graph against reference solutions using optimal transport, in real time. It flags loops, stalls, hallucinations, causal inversions, and off-track behavior as the agent runs, with sub-millisecond per-event latency.

The monitor operates in three information regimes:

• References + tool schemas: full structural alignment, pre-execution gate, and per-step diagnostics
• Schemas only: loop/stall detection plus prerequisite checks (no plan deviation)
• Bare stream: loop and stall detection via L1 vital signs

Files

Installation

pip install -r requirements.txt

The core algorithm (ontrack.py) only requires NumPy. The evaluation scripts additionally need sentence-transformers, datasets, scikit-learn, scipy, and optionally matplotlib for figures.

Quick start

Run the controlled test suite (no downloads needed)

python test_ontrack.py

This runs all 8 fault-injection scenarios (Table 2) with synthetic embeddings and prints PASS/FAIL for each.

Run the SWE-bench evaluation

python eval_swebench.py

This downloads sailplane/swe-agent-trajs (2,294 trajectories) from HuggingFace, embeds steps with all-MiniLM-L6-v2, and runs the full experiment suite:

• EXP 1: Resolved vs. unresolved discrimination (AUROC)
• EXP 1b: Length-controlled variant
• EXP A: Early warning and cost savings (abort threshold sweep)
• EXP B: L1-only exit-status detection
• EXP E: In-distribution splice injection
• EXP 2: Fault injection
• EXP 3: Ablation

Results are saved to a timestamped JSON file.

Run additional experiments

python eval_presubmit.py   # Table 5, Table 6, checkpoint ablation
python eval_reviewer.py    # Table 4, multi-seed robustness
python eval_review3.py     # Table 3 (L3 gate tests), AUPRC correction

Using OnTrack in your own project

import numpy as np
from ontrack import Config, Event, Reference, OnTrack

# Build a reference trajectory (known-good execution)
ref = Reference(
    action_embs=np.stack([...]),  # [m, d] action embeddings
    tools=["search", "read", "edit", "test", "commit"],
    parents=[[], [0], [0], [1, 2], [3]],  # DAG structure
    args_embs=np.stack([...]),    # [m, d] argument embeddings (optional)
    effect_embs=np.stack([...]),  # [m, d] observation embeddings (optional)
)

# Create evaluator
evaluator = OnTrack([ref], Config())

# Stream agent events one at a time
for step in agent_stream:
    event = Event(
        step_id=step.id,
        action_emb=embed(step.action),
        tool=step.tool_name,
        args_emb=embed(step.arguments),
        effect_emb=embed(step.result),
        irreversible=step.is_destructive,
        requires=step.required_artifacts,
    )
    result = evaluator.append_event(event)

    # result["verdict"] in: OK, EXPLORING, WARN, LOOP, STALLED,
    #                        OFF_TRACK, CAUSAL_INVERSION, BLOCK
    if result["verdict"] == "BLOCK":
        prevent_execution(step)
    elif result["verdict"] in ("LOOP", "OFF_TRACK"):
        alert_operator(result)

# At end of run
final = evaluator.finalize()
print(f"Final score: {final['final_score']:.3f}")

Key hyperparameters

lam is the main parameter to calibrate: it controls how aggressively the solver destroys mass for unmatched steps. The paper uses lam=0.1 on synthetic embeddings and lam=0.3 on SWE-bench with all-MiniLM-L6-v2.

Citation

@article{barazandeh2026ontrack,
  title={OnTrack: Real-Time Monitoring and Intervention in {LLM} Agent Trajectories via Streaming Structure-Aware Optimal Transport},
  author={Barazandeh, Babak and Swanson, Connor and Kulkarni, Chinmay and Mungel, Nikhil},
  journal={arXiv preprint arXiv:2610.12375},
  year={2026}
}

License

Apache 2.0