detector-scope-peer-review-llama
Detects Llama-written reviews of machine-learning papers (trained on the AI Peer Review Detection Benchmark, human reviews vs. Llama reviews). Gemma-2-2b fine-tuned as a review classifier, sparse autoencoders fine-tuned on it, and a sparse logistic-regression head over their features. Every prediction is a sum of contributions from sparse-autoencoder features, each with a validated natural-language description, so the detector can say which properties of the text drove its decision and where they occur. Paper: Building Auditable AI Text Detectors with Sparse Autoencoders; code: google-deepmind/detectorscope.
Use
pip install detectorscope
from detectorscope import AuditableDetector
det = AuditableDetector.from_pretrained("liamdugan/detector-scope-peer-review-llama")
out = det("The authors propose a novel framework for ...")
out.p_ai # probability that the text is AI-written
out.label # "ai" or "human" (logit threshold +0.08)
out.features # [Feature(id, importance, description, highlights), ...]
out.features lists every feature active in the text with a non-zero effect,
largest effect first. importance is how much the logit falls if that feature
is silenced: positive pushes toward AI, negative toward human, and
out.logit == out.bias + sum(f.importance for f in out.features). A list of
texts returns a list of detections.
To see where a feature fires:
out = det(text, return_highlights=True)
out.tokens # the text as a list of tokens
out.features[0].highlights # [(token index, activation), ...] of the top feature
The default head is logistic regression over the features whose
auto-interpretation score is at least 0.85. head="xgboost" or another
tau (0.00 to 0.95 in steps of 0.05), on from_pretrained or on the call,
selects one of the other 39 heads.
Performance
Default head; the decision threshold is fixed on the development data.
| split | AUROC | F1 | accuracy | documents |
|---|---|---|---|---|
| PRDB test, Llama reviews | 1.000 | 0.994 | 0.994 | 57,368 |
What it looks at
The features with the largest average contribution on the training data (weight × mean activation when active; density = share of the documents of each class in which the feature fires).
Toward AI
| feature | description | mean contribution | AI density | human density |
|---|---|---|---|---|
| 17_4373 | the word thorough | +0.993 | 75% | 9% |
| 13_12061 | numbered list items in parentheses within question sections of research papers | +0.898 | 26% | 0% |
| 7_5460 | the word one when it appears at the start of a sentence or bullet point | +0.745 | 50% | 3% |
| 13_164 | opening square bracket in list or string literals | +0.610 | 21% | 0% |
| 7_4720 | word written used to describe paper quality | +0.592 | 51% | 13% |
| 22_8578 | word firstly when used as a transition in academic paper reviews | +0.553 | 29% | 1% |
| 13_5375 | word firstly when used as a transition in academic or research paper critique sections | +0.519 | 30% | 2% |
| 7_156 | academic field noun when used in singular or plural form | +0.459 | 46% | 11% |
| 22_7140 | activates on the word potential | +0.446 | 67% | 14% |
| 7_2635 | academic paper review phrases like has several strengths or has several weaknesses | +0.446 | 23% | 0% |
Toward human
| feature | description | mean contribution | AI density | human density |
|---|---|---|---|---|
| 7_1616 | the word see used as a directive or reference to look elsewhere in academic reviews | -0.836 | 1% | 40% |
| 7_10672 | word seem or seems when used to express an opinion or observation about a paper or argument | -0.808 | 1% | 56% |
| 22_773 | closing parenthesis followed by a period or similar closing punctuation | -0.745 | 0% | 43% |
| 13_667 | polite requests in formal feedback or review contexts | -0.706 | 0% | 19% |
| 22_10620 | modal verb could in academic critique or feedback | -0.705 | 0% | 27% |
| 22_2854 | first person singular objective and reflexive pronouns | -0.629 | 1% | 33% |
| 7_10682 | the digit 3 when used in a numbered list or section reference | -0.569 | 38% | 74% |
| 7_1446 | word since appearing at the beginning of a sentence or clause | -0.487 | 0% | 24% |
| 17_16165 | modal verb will appearing in academic or technical discussion | -0.379 | 4% | 25% |
| 13_899 | past tense auxiliary verbs like were had and was | -0.368 | 3% | 29% |
Variants
Branches of this repo hold the paper's other two settings on the same data: ft-sae (the unmodified Gemma-2-2b with the fine-tuned SAEs) and base-sae (the unmodified Gemma-2-2b with the Gemma Scope SAEs). Load one with revision="ft-sae" or revision="base-sae". The general-domain detector is detector-scope.
Licence
The language model under extractor/ is Gemma-2-2b fine-tuned on these reviews, provided under
and subject to the Gemma Terms of Use
(see NOTICE). The sparse autoencoders derive from
Gemma Scope (CC-BY-4.0);
the heads and feature descriptions are CC-BY-4.0.
Citation
@misc{dugan2026detectorscope,
title={Building Auditable AI Text Detectors with Sparse Autoencoders},
author={Liam Dugan and Callum McDougall and Matthew Denton and Phillip Pham and Christine Kaeser-Chen and Chris Callison-Burch},
year={2026},
eprint={2610.30006},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.30006},
}