RealStop · local judge
A yes/no decision model—not a chatbot. This is the local judge used by RealStop's Claude Code Stop hook. It scores attempted turn endings so the hook can catch work handed back to you.
See the demo · Code & installation · Results · Limits
uv tool install realstop
realstop setup --judge local
realstop serve
Apple silicon is required for the supported local server. Keep realstop serve running in a separate terminal. Until it answers, the hook uses phrase rules. This model serves the Claude Code integration. RealStop 0.2.0 and later also include a Codex CLI adapter, which uses phrase rules only and does not use this model.
Recorded hook decisions on synthetic fixtures using the phrase rules; this animation is not a model benchmark or a live task-completion claim. Static explanation.
| At a glance | Scope |
|---|---|
| 67.5% hand-backs caught | Default local setting on 864 development-used test turn ends |
| 15.8% false interventions | Right stops incorrectly blocked on that same split |
| 27 ms median model time | Warm on an Apple M5 Pro; not cold-start or full-hook latency |
| Apache-2.0 | Code and model; research preview |
Approval enforcement belongs in the harness. RealStop cannot verify that claimed work was actually completed.
A real Claude Code session in a sandbox repo (RealStop 0.1.0, 2026-10-31). The agent ends with an offer (“Tell me when you want me to take it”), the Stop hook blocks the stop, and the agent re-checks the request and ends correctly without starting unrequested work. Recorded with an earlier wording of the hook message.
About this model
RealStop is an open-source Stop hook for Claude Code, released as a research preview. It is built to block hand-backs and let real approval stops through. A hand-back (RealStop calls it a quit) is a turn that ends by handing back work the agent could have done. When an agent ends its turn, RealStop asks one yes/no question: should the agent have kept working? If yes, the hook sends the agent back to work and still tells it to ask before it deploys, sends, spends or deletes. A pattern list defines which steps need the owner, so harness permissions stay on.
This model is RealStop's default judge. It runs on your computer. In local mode, RealStop sends no prompt or reply text to an outside judge. Claude Code itself still talks to its provider as usual. On the 864-row test split, which guided development, at its default (moderate) strength it caught 67.5% (195 of 289) of hand-backs at 15.8% (91 of 575) false interventions, against 56.7% (164 of 289) at 14.6% (84 of 575) for the phrase rules on the same rows. The installer asks which judge to use, and option 1 is this model. You can pick the cloud decision model instead: run realstop judge jev (or --judge jev at install) and give your key when setup asks. It is paid per use. Each judged stop sends the end of your request, the start and end of the reply, the tool names and up to 8 learned examples to the cloud decision model's provider, unredacted. We make no quality claim about it. The phrase rules and the skill are always on, as the base and the backup.
The model
- What it is. DecisionTune 1.0 with a LoRA on the attention layers and a one-unit head. The LoRA and head are 10.5 MB. The merged export in this repo is about 1.5 GB. The base model is decision-tune/decisiontune-1.0 (Apache-2.0). This repo holds only the pack.
- Output. The model is an encoder with a one-unit head and no generative head. It outputs one yes/no score per stop, so it cannot emit text.
- Input.
request:+ the owner's request (first 600 characters) +reply ending:+ the last 1,500 characters of the reply + the tool-call count + whether background work is running. The model answers one yes/no question. The hook corrects P(yes) to the real class mix and compares it with the threshold for the chosen strength (light, moderate or hard). The thresholds ship with this model. Thresholds from another model are badly wrong for it. - Speed and size. On an Apple M5 Pro (macOS 27.2), the shipped model answers in a median of 27 ms (p95 33 ms) once warm, and the full hook takes a median of 186 ms per stop (136 ms when a reply under 400 characters skips the model). The server runtime (
decision-tune[mlx]) is about 1 GB. There is no judge API fee. - Install. RealStop's
install.shfetches the pack from here and checks every file's sha256 against hashes pinned in the RealStop repo (see MODEL.md there).
Data
- 3,640 turn ends from 133 real sessions (127 Claude Code, 6 Codex CLI) of one developer. The text was redacted before labeling and training.
- Claude models made the labels. 34.4% (1,253 of 3,640) are hand-backs. In a blind check of 30 items by the developer, the labels agreed on 78% (21 of 27) of the items that could be decided. Kappa was 0.55.
- Under that developer's rules, an offer of more work after a finished answer counts as a hand-back, unless the offered step needs the owner's approval.
- Splits: train 1,805, dev 277, select 694, test 864. Test sessions are separate from the others. Dev and select share sessions with train (274 of 277 dev rows and 685 of 694 select rows).
- The training data is not released.
Results
Recall is the share of hand-backs blocked. False interventions (wrong nudges) are the share of right stops blocked. The results are offline replays of real turn ends.
Test split, paired. We scored this model at each strength and the phrase rules through the same hook on the same 864 rows (289 hand-backs, 575 right stops) from 21 sessions.
| judge | recall | false interventions | leaves the computer |
|---|---|---|---|
| This model, light | 53.6% (155 of 289) | 9.7% (56 of 575) | no |
| This model, moderate (default) | 67.5% (195 of 289) | 15.8% (91 of 575) | no |
| This model, hard | 75.4% (218 of 289) | 22.8% (131 of 575) | no |
| Phrase rules | 56.7% (164 of 289) | 14.6% (84 of 575) | no |
- Against the phrase rules, same rows. This model (moderate) caught more hand-backs: +10.7 points (95% interval +4.4 to +14.9), 67.5% (195 of 289) against 56.7% (164 of 289). False interventions were 15.8% (91 of 575) against 14.6% (84 of 575): +1.2 points (interval -3.3 to +5.4), no detectable difference. Intervals resample whole sessions.
- Strength settings. The light, moderate and hard settings were tuned on the select split only, to wrongly block about 5%, 10% and 15% of right stops. In practice each blocks somewhat more: on test, 9.7%, 15.8% and 22.8%. Moderate (the default) and hard trade more false interventions for higher recall, as the table shows.
Synthetic pairs. The model is weak on the public synthetic set of RealStopBench. It got both members of a pair right (a hand-back and its right-stop twin) in 16% (37 of 229) of pairs. It caught 31.4% of hand-backs at 15.3% false interventions. The phrase rules scored about the same: 32.3% recall at 16.2% false interventions. The model does not beat them there.
Approval-first nudge. RealStop's block message states the approval rule first. In two small retests with two revised message variants, attempts at approval-gated actions were 0 of 32, against 29 of 36 for the old message. Zero observed attempts is not a guarantee. Approval enforcement belongs in the harness.
Learning from your corrections
RealStop's realstop learn walks you through the corrections you gave the agent, one by one, and you confirm each with y or n. It needs a terminal and refuses when agent environment indicators are present. These checks are not proof that a person is present, and they are not a security boundary. Confirmed examples refit a personal copy of this model's 4 KB head on your computer, held close to the shipped head. RealStop uses the update only if its score on the public synthetic pairs drops by no more than 1 point, and the threshold can move only a small, bounded step. --list shows what was learned, --undo removes one example, and --reset removes everything learned. The files in this repo never change. How much it helps on real sessions has not been measured yet.
Limits
- All data come from one developer's sessions, and that developer's rules define a hand-back. Results may not hold for you.
- Labels are model-made. One small blind human check is the only measure of their agreement with a person.
- The test split's labels informed earlier work, so it is not a fresh holdout. On a sealed held-out set, an earlier version of this model showed no detectable difference from the phrase rules.
- The model reads the request, the end of the reply, a tool-call count and a background-work flag, and the hook's rules read tool results for denials. RealStop cannot check whether claimed work was done, so a bare "Done." can pass.
- In live runs on Claude Opus 5.5 and Claude Sonnet 5.5, RealStop with the phrase rules as judge removed closing offers. The pilots were too small to show that agents finish more work.
- English only. Short replies score low, so RealStop sends replies under 400 characters to the phrase rules.
- No result compares RealStop with simpler options (plain instructions or Anthropic's
ralph-loopplugin).
License
Apache-2.0 for the code and the model, the same license as DecisionTune.