AI Tracker Bot Classifier

A 396M-parameter encoder (fine-tuned ModernBERT-large) that decides whether a new-model alert from AI Tracker (@aitrackerbot) is a real new model or leak worth posting or a false alarm. It runs as INT8 ONNX on the bot's Raspberry Pi 5.

The bot watches API catalogs, docs and pricing pages, web-app bundles, LM Arena, official X accounts, news sitemaps, Hugging Face orgs and code repos. When an ID it has never seen appears, it wants to post "New model: X". The bot's ledger rules already stop exact repeats. The mistakes that got through those rules look like this:

• labels that aren't models (grok-voice is a call-history label, nemotron_v3 a reasoning parser, codex_apps a tool)
• routes and aliases of a model it already knew (muse-spark-1.3-(max, claude-fable-5.1-search, grok-45)
• old models resurfacing through a new source (grok-4.20-fast in a bundle backfill, dated Gemini 2.5 previews re-added by pagination)
• research artifacts and fixtures on official Hugging Face orgs (nvidia/mvaoi3d-lora, spec-decoding-subfolder-fixture)
• editorial slugs (grok-build-for-everyone)

This model reads one candidate at a time, with the context the bot has, and returns P(false_alarm).

Estimated on the bot's real history (held-out cross-validation), at the deployed threshold 0.96:

• It stops 7 of the 14 real false alarms that the current ledger rules miss.
• It holds back 1 of 68 real posts: an ambiguous Arena sighting of gemini-3.5-pro at a time when Gemini 3.6–3.8 Flash were already known.
• Across all 29 false alarms in the record, rules plus classifier would have stopped about 22 (76%), against 15 (52%) for the rules alone.

The real sample is small, so treat these numbers as estimates (details and caveats below).

How the bot uses it

• The tracker renders the input text itself (training/tracker/alertClassifier.js, input format v2). A loopback service on the Pi (training/pi-service/serve.py) scores it with onnx/model_int8.onnx.
• A candidate is held back only when P(false_alarm) >= 0.96. Held-back alerts are not posted to X, Discord, email or Reddit. The operator gets a Discord DM with the score and a link to the event, so a wrongly held-back model is never silently lost.
• The check fails open. If the service is down, errors or times out, the bot posts exactly as before.
• Operator-verified posts skip the check.

Input format (v2)

One input per candidate, about 500 tokens (99th percentile 770; truncated at 1024). It contains:

• source (name, id, type, host), stage (leak/release) and flags, detection date
• the candidate ID, the maker and the ledger key the bot inferred
• candidate history: earlier ledger records under that same key, for example a leak seen weeks before the official release
• catalog details, if the source has them
• other IDs added or removed in the same change
• the 8 most similar models already in the bot's ledger and the 4 highest-versioned ones in the family, each with stage and first-seen date
• the diff lines around the candidate

Example (a real false alarm from September 2026, P(false_alarm) 0.99):

[source] Grok web app bundle models | id grok-web-app-models | type app-bundle-models | host grok.com | topic Grok/xAI
[signal] stage leak | official yes | code reference only no | official preview no | access -
[detected] 2026-09-25
[candidate] grok-voice | maker xAI | key xai:grok-voice
[candidate history] none
[details] none
[also added] none
[removed] none
[summary] Grok/xAI model leak: grok-voice
[known similar] grok-voice-transcribe-2.0 (leak 2026-09-18); grok-voice-agent-builder (release 2026-08-09, leak 2026-08-09); grok-voice-think-fast-1 (release 2026-08-09); grok-voice-think-fast-2 (release 2026-08-09); grok-voice-think-fast-1.0 (release 2026-07-30); grok-voice-think-fast-2.0 (release 2026-07-30); grok-vapi (release 2026-08-09); grok-4.7-reasoning (release 2026-09-21)
[newest in family] grok-4.20-0309-v2 (leak 2026-09-19); grok-4.20-0309 (release 2026-08-10); grok-4.20-0309-non-reasoning (release 2026-08-10); grok-4.20-0309-reasoning (release 2026-08-10)
[evidence]
     "grok-4.5",
     "grok-4.6",
     "grok-4.7",
+    "grok-voice",
     "grok-voice-agent-builder",
     "grok-voice-transcribe-2.0"
   ]

Labels: 0 = false_alarm, 1 = post.

Results

Real bot decisions, held out

The real set is 102 candidates the bot actually announced between 2026-07-21 and 2026-09-27. Labels come from what the operator kept versus deleted, and from the recorded deletion reasons. Late-but-real models count as posts.

20 of those candidates (15 false alarms, 5 duplicate posts) are already blocked by today's ledger rules, so the classifier never sees them in production. The honest test set is therefore the other 82 (68 posts, 14 false alarms).

The estimates come from 5-fold cross-validation. Each fold trains on the synthetic data plus 4/5 of the real set and scores the held-out fifth. The final published model was then trained the same way on all 82.

A lower threshold stops more false alarms: at 0.875 the published recipe stops 12 of 14 but holds back 2 posts. 0.96 was chosen because missing a real model is worse than one extra post.

The published model's own scores on these 82 are in-sample (11/14 stopped, 0/68 held back) and are not an estimate of future accuracy.

Synthetic validation (in distribution)

468 held-out teacher-verified scenarios (272 posts, 196 false alarms): AUC 0.961. At 0.96 it stops 114/196 false alarms (58%) and holds back 3/272 posts (1.1%).

INT8 vs full precision

onnx/model_int8.onnx is weight-only INT8: ONNX Runtime MatMulNBits, 8-bit, symmetric, block size 32, accuracy_level=4. It matches the fp32 model:

Plain dynamic W8A8 quantization (quantize_dynamic) does not work for this model. It quantizes every MatMul input with one scale per tensor, and ModernBERT's MLP down-projection inputs carry outlier channels. Measured on synthetic val + real (570 inputs):

Speed

• Raspberry Pi 5 (8 GB), shared with about 40 other services (load average 3–6 during the test), 3 threads, ~530-token inputs: 5.9 s median, 7.1 s max per candidate. Load takes 4 s, RSS is about 1.0 GB.
• x86 workstation CPU, 8 threads: 0.22 s.

The check only runs on candidates that survive the ledger rules (a few per day), so the delay is a few seconds before a post. ModernBERT-base would be about 3x faster but ranked real alerts clearly worse (table above).

Training data

• Synthetic, teacher-written and teacher-verified: 4,636 scenarios (4,168 train + 468 val; 2,696 post / 1,940 false alarm) across 27 categories. The teacher was Qwen3.8-Flash-Next (NVFP4, SGLang) with thinking on at low effort.
  • Specs are sampled from the tracker's real world (source types, makers, families, ledger contents) and dates up to the end of 2027. Later batches are sampled by (source type × label) cell so each source's label mix is realistic: Arena and bundles mostly carry false alarms, catalogs and docs mostly real models.
  • The teacher writes a scenario: the source's diff lines, ledger history and the label.
  • The tracker's own renderer turns each scenario into the exact production input.
  • A blind judge (same teacher, thinking) labels it. Where the judge disagrees with the intended label, an adjudicator checks whether that label is supported by the visible input alone. 4,540 kept on agreement, 96 kept by the adjudicator, 309 dropped.
• Real: the 82 non-blocked candidates above, weight 1. Synthetic rows are reweighted so each source type's post rate moves toward the real one (shrunk to 0.5 with k=10).
• data/train.jsonl and data/val.jsonl hold the synthetic data. The real set is not published.

Lessons that shaped the recipe:

• The first synthetic set gave details and leak stage to any source. The model learned "has details and is a leak → post", which is backwards for real Arena and bundle alerts. Real-gold AUC was 0.33 until events carried the source's real stage and detail fields.
• Hiding the candidate's own earlier leak record made official releases of leaked models look brand-new. Showing it (the candidate history line) plus a release-after-leak category fixed that.
• Real-data weight 1 beat 2 and 4. Higher weights overfit per-source quirks of the 82 examples.
• Weight soups need a shared head init. Soups of seeds with different head inits squash every probability below 0.8.

Files

Usage

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo = "ProCreations/ai-tracker-bot-classifier"
tok = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))
tok.enable_truncation(max_length=1024)
session = ort.InferenceSession(hf_hub_download(repo, "onnx/model_int8.onnx"))

ids = np.array([tok.encode(text).ids], dtype=np.int64)  # text rendered by alertClassifier.js (format v2)
logits = session.run(None, {"input_ids": ids, "attention_mask": np.ones_like(ids)})[0][0]
p = np.exp(logits - logits.max()); p /= p.sum()
print({"false_alarm": float(p[0]), "post": float(p[1])})

With transformers: AutoModelForSequenceClassification.from_pretrained(repo).

Limitations

• The real held-out test is small: 14 false alarms and 68 posts. "7 of 14" could easily be 5 or 9 on a different sample, and the scores varied noticeably between training seeds.
• Real labels come from one operator's keep/delete decisions.
• A few real cases are genuinely ambiguous, such as the held-back gemini-3.5-pro Arena sighting. The DM is the backstop for these.
• The model is specific to this bot and its input format. New source types or naming styles can drift away from what it learned. Fail-open and the DM trail mean drift shows up as extra posts or reviewable DMs, not silent losses.
• All synthetic data comes from one teacher model, so its blind spots may carry over.