detector-scope-peer-review-gpt4o
Detects GPT-4o-written reviews of machine-learning papers (trained on the AI Peer Review Detection Benchmark, human reviews vs. GPT-4o reviews). Gemma-2-2b fine-tuned as a review classifier, sparse autoencoders fine-tuned on it, and a sparse logistic-regression head over their features. Every prediction is a sum of contributions from sparse-autoencoder features, each with a validated natural-language description, so the detector can say which properties of the text drove its decision and where they occur. Paper: Building Auditable AI Text Detectors with Sparse Autoencoders; code: google-deepmind/detectorscope.
Use
pip install detectorscope
from detectorscope import AuditableDetector
det = AuditableDetector.from_pretrained("liamdugan/detector-scope-peer-review-gpt4o")
out = det("The authors propose a novel framework for ...")
out.p_ai # probability that the text is AI-written
out.label # "ai" or "human" (logit threshold +0.29)
out.features # [Feature(id, importance, description, highlights), ...]
out.features lists every feature active in the text with a non-zero effect,
largest effect first. importance is how much the logit falls if that feature
is silenced: positive pushes toward AI, negative toward human, and
out.logit == out.bias + sum(f.importance for f in out.features). A list of
texts returns a list of detections.
To see where a feature fires:
out = det(text, return_highlights=True)
out.tokens # the text as a list of tokens
out.features[0].highlights # [(token index, activation), ...] of the top feature
The default head is logistic regression over the features whose
auto-interpretation score is at least 0.85. head="xgboost" or another
tau (0.00 to 0.95 in steps of 0.05), on from_pretrained or on the call,
selects one of the other 39 heads.
Performance
Default head; the decision threshold is fixed on the development data.
| split | AUROC | F1 | accuracy | documents |
|---|---|---|---|---|
| PRDB test, GPT-4o reviews | 1.000 | 0.998 | 0.998 | 57,368 |
What it looks at
The features with the largest average contribution on the training data (weight × mean activation when active; density = share of the documents of each class in which the feature fires).
Toward AI
| feature | description | mean contribution | AI density | human density |
|---|---|---|---|---|
| 13_15950 | academic paper review questions asking for more information details or clarification | +1.002 | 75% | 1% |
| 17_8459 | academic review feedback or peer review decisions | +0.814 | 91% | 0% |
| 22_4800 | double colons following bolded markdown section headers | +0.672 | 51% | 0% |
| 17_12216 | activates on the word potential | +0.515 | 87% | 14% |
| 7_10900 | the word particularly when used as an adverb to specify or emphasize a context | +0.438 | 73% | 8% |
| 17_11188 | academic research terminology related to study design and methods | +0.388 | 62% | 7% |
| 7_4872 | activates on the word potentially often used to hedge academic critique or discuss future implications | +0.363 | 41% | 1% |
| 13_15079 | societal impact in research paper sections | +0.357 | 77% | 0% |
| 22_16295 | the word beyond in academic or technical contexts | +0.320 | 47% | 7% |
| 7_5712 | forms of the word enhance such as enhances enhancing and enhancements | +0.307 | 70% | 9% |
Toward human
| feature | description | mean contribution | AI density | human density |
|---|---|---|---|---|
| 13_5785 | first person singular pronoun i at start of sentences or after punctuation | -0.919 | 10% | 73% |
| 17_13291 | forms of the verb seem | -0.639 | 9% | 49% |
| 22_8699 | academic review section headers like clarity and clarify | -0.574 | 1% | 35% |
| 7_6962 | word for at the start of a sentence | -0.427 | 6% | 35% |
| 7_13654 | math and variable notation indices and subscripts | -0.392 | 2% | 38% |
| 7_7454 | the adverb very in various forms and cases | -0.341 | 11% | 45% |
| 13_5925 | modal verb might | -0.334 | 4% | 20% |
| 7_23 | academic document figure or fig references | -0.306 | 9% | 32% |
| 22_13623 | polite imperative requests used in academic review feedback | -0.297 | 0% | 20% |
| 13_12569 | words related to academic paper authors and roles like writer author or auth | -0.294 | 1% | 12% |
Variants
Branches of this repo hold the paper's other two settings on the same data: ft-sae (the unmodified Gemma-2-2b with the fine-tuned SAEs) and base-sae (the unmodified Gemma-2-2b with the Gemma Scope SAEs). Load one with revision="ft-sae" or revision="base-sae". The general-domain detector is detector-scope.
Licence
The language model under extractor/ is Gemma-2-2b fine-tuned on these reviews, provided under
and subject to the Gemma Terms of Use
(see NOTICE). The sparse autoencoders derive from
Gemma Scope (CC-BY-4.0);
the heads and feature descriptions are CC-BY-4.0.
Citation
@misc{dugan2026detectorscope,
title={Building Auditable AI Text Detectors with Sparse Autoencoders},
author={Liam Dugan and Callum McDougall and Matthew Denton and Phillip Pham and Christine Kaeser-Chen and Chris Callison-Burch},
year={2026},
eprint={2610.30006},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.30006},
}