Project Sidewalk Tagger: Crosswalk
A multi-label classifier that predicts which accessibility tags apply to a Crosswalk label in a crop of a Google Street View image. Project Sidewalk serves it through sidewalk-ai-api to suggest tags for crowdsourced labels.
This is the DINOv2 model trained on the cleaned dataset in Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset (ASSETS '24). Training and evaluation code is in sidewalk-tagger-ai.
Provenance
validated-dino-cls-b-crosswalk-tags-best.pth is a byte-for-byte copy of the file of the same name in
projectsidewalk/sidewalk-tagger-ai-models at commit 65959dbc80b87e4f39385204c4b639cbcf58e1a8,
which holds every model from the paper and remains available unchanged.
sha256: 2399313abc4b462fa9a708695f6febd05d0bd199e1f7f715af55ee312e618f55
Model
- Backbone: DINOv2 ViT-B/14 with 4 register tokens (
dinov2_vitb14_reg4), fine-tuned. - Head:
Linear(768, 256) → ReLU → Linear(256, 10), with a sigmoid per class. - Input: RGB crop resized to 256 px and padded to a multiple of 14, normalized with ImageNet mean and std.
- Format: a PyTorch state dict; it loads with
torch.load(..., weights_only=True).
The architecture is custom rather than a transformers class. See DinoVisionTransformerClassifier and
ImageTagger in sidewalk_ai_api/tagger.py
for loading and preprocessing.
Classes
The output has 10 classes, in this order. NULL entries are unused placeholders that preserve the class indices from
training.
| Index | Tag |
|---|---|
| 0 | brick/cobblestone |
| 1 | broken surface |
| 2 | bumpy |
| 3 | NULL |
| 4 | NULL |
| 5 | paint fading |
| 6 | rail/tram track |
| 7 | NULL |
| 8 | uneven surface |
| 9 | very long crossing |
Thresholds
Sigmoid scores aren't calibrated probabilities. The per-tag thresholds Project Sidewalk uses to suggest that a tag is
present or absent are in
tag_thresholds.json in
sidewalk-ai-api.
Evaluation
See the paper for per-tag results on the test split of
sidewalk-tagger-ai-validated.
REPRODUCE_RESULTS.md in
sidewalk-tagger-ai reproduces them.
Citation
@inproceedings{10.1145/3663548.3688531,
author = {Liu, Xinlei and Wu, Kevin and Kulkarni, Minchu and Saugstad, Michael and Rapo, Peyton Anton and Freiburger, Jeremy and Hosseini, Maryam and Li, Chu and Froehlich, Jon E.},
title = {Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset},
year = {2024},
publisher = {Association for Computing Machinery},
doi = {10.1145/3663548.3688531},
booktitle = {Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility},
articleno = {103},
numpages = {12}
}