Signpost label-quality classifier (hashed backend)

Judges whether an accessibility label actually identifies the control it is attached to -- the question left over after axe-core and Xcode's Accessibility Inspector have both passed. There is a name; does the name mean anything on this screen?

Repository: NagaYu/signpost-classifier

It classifies one label into eight classes: ok plus seven ways a label fails (generic, role_echo, identifier, duplicate, verbose, missing, mismatch), and reports the character span of the offending part.

How to use it

from signpost.context import ContextBuilder, ContextConfig
from signpost.model import QualityClassifier

clf = QualityClassifier.load("path/to/this/checkpoint")
builder = ContextBuilder(ContextConfig(level="full"))  # must match the checkpoint
bundles = builder.build_screen(screen)               # a signpost.types.ScreenTree
for bundle, prediction in zip(bundles, clf.predict(bundles)):
    print(bundle.name, prediction.error_type, round(prediction.confidence, 3))

or from the command line, over an axe run, an XCUITest dump or a uiautomator dump:

signpost check tree.json --model path/to/this/checkpoint

The context level is pinned in the checkpoint and enforced on every call: feeding it bundles built at another level raises rather than returning numbers that look fine and mean nothing.

What was measured

Held-out origins -- no site or app on the test side appears in training -- with 860 labels, 309 planted defects and 511 labels the strict quality gate calls good.

The operating threshold is 0.644, chosen on the training split as the most talkative cut whose measured over-flagging stayed inside a 2% budget. The test numbers above were computed at that threshold and it was never tuned on them.

Per class

Does the context help?

The same backend, the same hyperparameters, the same seed and the same screens, trained once per context level. The only thing that varies is what the model was allowed to look at.

Training

• Corpus: signpost/data/seed_screens.jsonl (synthetic)
• Split: by origin, 30% of nodes held out, stratified by language; train and test share no origin, checked before training.
• Defects: planted by signpost/inject.py at a rate of 35%, with a different seed on each side of the split so a test defect is not a training defect relocated.
• Backend: hashed, class weighting balanced, seed 0.
• No base model. The hashed backend is a matrix of floats over hashed context features, trained from scratch. There is no encoder to attribute, which is why the frontmatter carries no base_model.
• Fit: 0.7 s on Darwin arm64 python 3.13.5.
• No rewrite adapter is included. The template rewriters in signpost.rewrite need no model and are what signpost check uses by default.

Limitations -- read these before quoting a number above

• The corpus is synthetic. Every screen was composed by scripts/make_seed_corpus.py from a vocabulary written for this project, and every defect was planted by a generator this project also wrote. A model evaluated here is partly being measured on how learnable that generator is. Real screens are messier: more abbreviations, more house style, more labels that are bad in ways nobody thought to plant. Treat these figures as an upper bound on a real codebase rather than an estimate of one.
• Some classes it is bad at. Weakest in this run: role_echo recall 0.19 on 48 rows; mismatch recall 0.32 on 53 rows. In the project's reference run (python benchmarks/run.py, seed 0) the two weakest are role_echo recall 0.21 and mismatch recall 0.32 -- and mismatch is the class this project exists for. A defect this model misses is a defect nobody is told about.
• It is not a replacement for anything. Not for axe-core, not for Xcode's Accessibility Inspector or Android's lint checks -- they are better than this model at the classes they cover, missing above all -- and not for review by people who use assistive technology. It narrows what a human has to look at.
• It does not know what anyone will experience. Every finding points at a documented guideline (WCAG 2.2, Apple's and Google's own guidance) through signpost.cite, and says nothing about how any person will perceive a screen.
• duplicate is a property of a screen, not of a label. A row taken out of its screen cannot be judged for that class.
• The threshold is a policy, not a fact. A team gating a merge queue can afford fewer false alarms than a team reviewing a hundred screens by hand; calibrate_threshold is how they pick their own, and set_threshold is how they impose it.

Reproducing this checkpoint

python scripts/train.py --backend hashed --seed 0 --test-frac 0.30 --inject-rate 0.35

Same seed, same bytes: the hashed backend trains full-batch with no sampling anywhere in the loop.

Licence

mit. The corpus is synthetic and carries no third-party redistribution question; signpost/collect.py is the path for real pages and keeps per-row provenance.