botest v1 (round 12)

Weights for botest, a local detector for machine-written text. It marks the sentences in a document that read as generated and reports a level for the document as a whole. Everything runs on the machine it is installed on; no text leaves it.

This repository holds the model only: the weights, the files the scorer needs to load them, and the calibration that decides what counts as a verdict. The scorer itself, the evaluation harness and the data-building scripts live with the source.

What it is

It is a token classifier, not a document classifier. Each token gets a level, the sentence score comes from its tokens and the document score from its sentences, which is what lets a report point at the passages rather than returning one number for the page.

Using it

botest model fetch          # pulls this repository and checks every file's sha256
botest report draft.txt     # the sentences that read as generated, and why
botest score draft.txt      # the same as JSON, with byte offsets

model.toml in this repository is both the download manifest and the descriptor the scorer loads, so the files that were fetched and the files that get loaded cannot disagree about what the model is.

The four files beside the weights are not optional: config.json and head.json describe the network and its outputs, tokenizer.json is the vocabulary, and calibration.json carries the per-language temperature and the three thresholds. Without the calibration the model produces logits and nothing is allowed to call them a verdict.

Which languages it will judge, and why only those

A language is judged only if calibration.json marks it calibrated, which means its thresholds were fitted on held-out text in that language and checked against a false-positive budget. The scorer refuses every other language by name rather than returning a number it cannot stand behind:

$ botest score japanese.txt
"This version does not assess Japanese. Supported: Dutch, English, French, German,
 Italian, Polish, Portuguese, Spanish and Turkish."

Japanese, Chinese and Russian were trained and measured and are withheld: the model scores them, the measurement did not reach the bar the other nine did. The refusal is read from the calibration file at run time, not from a list compiled into the binary, so a language cannot be scored and documented as refused at the same time.

What it was measured on

Every number this project publishes comes from frozen held-out sets that no training round may touch, and is reproduced by one script:

eval/gates_langs.sh <model dir> <out dir>

The held-out sets are built from human text that predates the generators (arXiv 2021, Stack Exchange before 2022, encyclopaedia articles, essay-mill prose) plus generated rewrites of those same sources, by family: 16 open-weight models, and the GPT, Claude and Grok families. A document's leakage group hashes deterministically to one split, so a source cannot appear on both sides of a measurement.

The per-language and per-family table lands here as EVAL.md, generated by eval/make_eval_md.py from one run of the script above on these exact fp16 weights under this exact calibration. It is not here yet: the run is in progress, and the generator refuses to print a combined table whose languages were not all measured by the same model and the same calibration, which is a bar a half-finished run does not clear. Until it appears, this card publishes no accuracy number at all rather than a provisional one.

One thing will matter more than any cell in that table: every generated document in those sets is a model answering our prompt, and a model doing a real person's specific errand is a harder case than any of them.

What it is not

Not a plagiarism checker, not a proof of authorship, and not evidence. It reports what it measured, with the limits of that measurement attached, and a score near a threshold means the text is near a threshold — not that a person did something.

Reproducing the weights

The training round that produced them, the data recipe and the calibration fit are scripts in the source tree (training/hf_jobs/launch_round.sh, training/src/botest_training/). Each round's bars are pre-registered before it runs, which is the only way a gate result afterwards means anything.

License

Apache-2.0, as the code is. The base encoder jhu-clsp/mmBERT-small is MIT.