sidewalk-validator-ai-crosswalk
A binary image classifier that checks one crowdsourced Project Sidewalk Crosswalk label: given a crop of street-level imagery centred on the label, it predicts whether the label is correct (the image shows a crosswalk) or incorrect. It is facebook/dinov2-large fine-tuned with a two-class head (id2label = {0: correct, 1: incorrect}) by sidewalk-validator-ai.
Intended use
Automatic validation of new Crosswalk labels in Project Sidewalk. sidewalk-ai-api rebuilds the crop from the panorama and label point, runs this model, and looks the verdict's confidence up in confidence_precision_map.json. SidewalkWebpage records the verdict as an Agree/Disagree vote by the SidewalkAI account only when that estimated accuracy is at least 0.92 (ai-validation-min-accuracy); otherwise it is stored as Unsure. The model is one input to label quality, not a replacement for human validation.
Out of scope: finding features in whole panoramas (that is RampNet's job), other label types, and imagery that is not a label-centred crop made the same way as the training crops.
Training data
projectsidewalk/sidewalk-validator-ai-dataset-crosswalk (content hash TODO: dataset not staged with this model; copy the hash from the dataset card). TODO: the linked dataset repo holds an earlier selection; whether this model's training data is published, which includes deployments that are not public, is a policy decision (issues #2 and #7). Labels are taken as correct or incorrect from human validation votes only (AI votes excluded); see the dataset card for the exact thresholds. Split sizes: train 2,089, val 448, test 445, test_heldout 138 (tucson-az and 2 non-public deployments). The split is grouped by panorama (no panorama in two splits), stratified by class, seeded, with optional whole held-out cities (split_data.py).
How the input crop is made
The panorama is resized to 8192×4096. A 2048² perspective view with a 90° field of view is rendered centred on the label (sidewalk-panorama-tools), with rig-tilt correction from the label's camera pitch and roll (beta = 0.95). Depth Anything V2 (metric, VKITTI, ViT-B) estimates the depth at the view centre; the crop half-width is 6100 / depth_m pixels, clipped to the view, and the crop's long side is resized to 640 px (training crops are stored as WebP quality 95).
Deployment order. The model expects tilt-corrected crops, so the serving API must apply the same correction (sidewalk-ai-api#17).
Preprocessing and training
Inference uses this repo's preprocessor_config.json (BitImageProcessor: resize shortest edge 256, centre-crop 224, ImageNet mean/std); load it with use_fast=False to match the API's PIL path. Training adds RandomResizedCrop(224, scale=(0.7, 1.0)) and horizontal flips; validation uses the processor itself. Hugging Face Trainer, learning rate 1e-5, batch 8 × 4 gradient accumulation, 10% warm-up, 25 epochs, seed 42. The published weights are the checkpoint with the best validation macro_f1 (0.959; checkpoint-528); trainer_state.json keeps the full log history.
from PIL import Image
from transformers import AutoImageProcessor, AutoModelForImageClassification
processor = AutoImageProcessor.from_pretrained("projectsidewalk/sidewalk-validator-ai-crosswalk", use_fast=False)
model = AutoModelForImageClassification.from_pretrained("projectsidewalk/sidewalk-validator-ai-crosswalk").eval()
probs = model(**processor(Image.open("crop.webp").convert("RGB"), return_tensors="pt")).logits.softmax(-1)[0]
verdict = model.config.id2label[int(probs.argmax())] # "correct" or "incorrect"
Evaluation
From metrics.json (evaluate.py), on the best checkpoint. The verdict is the argmax with ties going to correct, as in the API.
| slice | n (correct / incorrect) | accuracy | balanced accuracy | AUC | macro-F1 |
|---|---|---|---|---|---|
| test (cities seen in training) | 445 (247 / 198) | 0.957 | 0.954 | 0.980 | 0.957 |
| test_heldout (held-out cities) | 138 (65 / 73) | 0.906 | 0.907 | 0.941 | 0.906 |
| hard (test + test_heldout rows the deployed model got wrong) | 5 (4 / 1) | 0.600 | 0.750 | 0.750 | 0.583 |
Per class:
| slice | class | precision | recall | F1 | support |
|---|---|---|---|---|---|
| test | correct | 0.942 | 0.984 | 0.962 | 247 |
| test | incorrect | 0.979 | 0.924 | 0.951 | 198 |
| test_heldout | correct | 0.882 | 0.923 | 0.902 | 65 |
| test_heldout | incorrect | 0.929 | 0.890 | 0.909 | 73 |
| hard | correct | 1.000 | 0.500 | 0.667 | 4 |
| hard | incorrect | 0.333 | 1.000 | 0.500 | 1 |
Recall at precision on the test split (the score threshold that achieves it in brackets):
| class | P ≥ 0.90 | P ≥ 0.92 | P ≥ 0.95 | P ≥ 0.99 |
|---|---|---|---|---|
| correct | 1.000 (≥ 0.0045) | 0.992 (≥ 0.1248) | 0.955 (≥ 0.9987) | 0.057 (≥ 1.0000) |
| incorrect | 0.955 (≥ 0.0004) | 0.944 (≥ 0.0011) | 0.934 (≥ 0.0068) | 0.889 (≥ 0.9813) |
Calibration
confidence_precision_map.json maps the model's softmax confidence for its verdict to the estimated probability that the verdict is right. It is fitted by calibration/calibrate.py with isotonic regression, reweighted from the training class mix to the deployment class prior, and is a drop-in for sidewalk-ai-api/accuracy_mappings/crosswalk.json.
Lowest confidence at which the map reaches the 0.92 gate:
| verdict | confidence at the gate |
|---|---|
| correct | 0.9925 |
| incorrect | never reaches the gate |
Deployment prior: correct 0.939, incorrect 0.061 (from rawLabels of amsterdam, auckland, bayonne-fr, blackhawk-hills-il, burnaby, cdmx, chandigarh-india, chicago-il, cliffside-park-nj, clifton-nj, columbus-oh, cuenca, danville-il, detroit-mi, fort-wayne-in, gainesville-fl, hackensack-nj, houston-tx, knox-oh, la-ca, la-piedad, laurens-ia, madison-wi, maywood-nj, mendota-il, newberg-or, newport-ky, niagara-falls-ny, oradell-nj, paterson-nj, pittsburgh-pa, rancagua-chile, richmond-va, santiago-chile, sao-paulo-brazil, seattle-wa, spgg, st-louis-mo, teaneck-nj, tucson-az, vancouver-wa, virden-il, walla-walla-wa, waltham-ma, washington-dc, west-chester-pa, human votes only (and 8 non-public deployments), measured 2026-10-20). Fitted on 448 val rows; evaluated on 445 test rows, reweighted to the same prior.
Overall expected calibration error (ECE): 0.0539.
| verdict | n | weighted share | precision | ECE | coverage at 0.92 | precision at 0.92 |
|---|---|---|---|---|---|---|
| correct | 258 | 0.928 | 0.995 | 0.0389 | 0.991 | 0.995 |
| incorrect | 187 | 0.072 | 0.788 | 0.2579 | 0.000 | n/a |
Held back: the incorrect map is capped at 0.9199: every threshold whose fitted accuracy is higher serves 0.9199 instead, so that verdict never reaches the 0.92 gate and never becomes a vote. Reason: first ship of a retrained revision: the deployment prior is under review (issue #56) and the revision has no production outcomes to fit on yet.
Uncapped, incorrect would have had coverage 0.422 at precision 1.000 on the evaluation split (n 98).
The map is fitted on the validation split, which also selected the checkpoint, so it is mildly optimistic.
Known failure modes and limitations
- Trained and evaluated only on Google Street View panoramas from the cities listed in the dataset card; performance elsewhere, and on other imagery, is unmeasured beyond the held-out cities above.
- Test metrics are on the training class mix (about 55:45 correct:incorrect). Production traffic is mostly correct labels, so precision on
incorrectverdicts is lower in deployment than the test table suggests; the calibrated map accounts for this, the raw metrics do not. - The
hardslice is small; read its numbers with wide error bars. - Correct/incorrect comes from crowd votes, which are themselves noisy; only decisively voted labels are used, so borderline cases are under-represented.
- A crop made without the tilt correction, or with a different depth model, is out of distribution.
- TODO: per-label-type failure modes from a review of this model's errors (e.g. the hard-example list); add them as
failure_modesin card_overrides.json.
Files
| file | contents |
|---|---|
model.safetensors, config.json |
weights and Dinov2ForImageClassification config |
preprocessor_config.json |
the image processor used in validation, evaluation and serving |
confidence_precision_map.json |
confidence → estimated accuracy, per verdict (the API's format) |
calibration_report.json |
prior, ECE and gate coverage of that map (when present) |
metrics.json |
the evaluation above |
trainer_state.json |
trimmed: log history and best checkpoint only |
training_run.json |
seed, epochs, package versions, generation settings (when present) |
Lineage
| year | work | relation |
|---|---|---|
| 2019 | Weld et al., ASSETS'19 (doi) | First deep-learning auto-validation of Project Sidewalk labels on depth-sized crops. |
| 2022 | Duan et al., ASSETS'22 (doi) | Validated training data and cross-city transfer; the direct ancestor of this pipeline. |
| 2023 | Oquab et al., DINOv2 (arXiv) | The pretrained backbone. |
| 2024 | Yang et al., Depth Anything V2 (code) | Metric depth that sizes each crop. |
Published from sidewalk-validator-ai at commit e313251c2571cb2a585835f812eaac85b851b7b4 with publish_to_hf.py.
Citation
No DOI has been minted for this model yet.
There is no paper for this model yet. Please cite Project Sidewalk and link the training repository:
@inproceedings{saha2019projectsidewalk,
author = {Saha, Manaswi and Saugstad, Michael and Maddali, Hanuma Teja and Zeng, Aileen and Holland, Ryan and Bower, Steven and Dash, Aditya and Chen, Sage and Li, Anthony and Hara, Kotaro and Froehlich, Jon},
title = {Project Sidewalk: A Web-based Crowdsourcing Tool for Collecting Sidewalk Accessibility Data at Scale},
booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
year = {2019},
doi = {10.1145/3290605.3300292}
}