Project Sidewalk Tagger: Obstacle
A multi-label classifier that predicts which accessibility tags apply to a Obstacle label in a crop of a Google Street View image. Project Sidewalk serves it through sidewalk-ai-api to suggest tags for crowdsourced labels.
This is the DINOv2 model trained on the cleaned dataset in Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset (ASSETS '24). Training and evaluation code is in sidewalk-tagger-ai.
Provenance
validated-dino-cls-b-obstacle-tags-best.pth is a byte-for-byte copy of the file of the same name in
projectsidewalk/sidewalk-tagger-ai-models at commit 65959dbc80b87e4f39385204c4b639cbcf58e1a8,
which holds every model from the paper and remains available unchanged.
sha256: da4f7ae0bb21d1a5b13a785ebc8553dcd8746eb823183f5c53332fe078009b6d
Model
- Backbone: DINOv2 ViT-B/14 with 4 register tokens (
dinov2_vitb14_reg4), fine-tuned. - Head:
Linear(768, 256) → ReLU → Linear(256, 20), with a sigmoid per class. - Input: RGB crop resized to 256 px and padded to a multiple of 14, normalized with ImageNet mean and std.
- Format: a PyTorch state dict; it loads with
torch.load(..., weights_only=True).
The architecture is custom rather than a transformers class. See DinoVisionTransformerClassifier and
ImageTagger in sidewalk_ai_api/tagger.py
for loading and preprocessing.
Classes
The output has 20 classes, in this order. NULL entries are unused placeholders that preserve the class indices from
training.
| Index | Tag |
|---|---|
| 0 | construction |
| 1 | fire hydrant |
| 2 | NULL |
| 3 | height difference |
| 4 | litter/garbage |
| 5 | narrow |
| 6 | NULL |
| 7 | parked bike |
| 8 | parked car |
| 9 | parked scooter/motorcycle |
| 10 | pole |
| 11 | sign |
| 12 | stairs |
| 13 | NULL |
| 14 | trash/recycling can |
| 15 | tree |
| 16 | vegetation |
| 17 | mailbox |
| 18 | seating |
| 19 | uneven/slanted |
Thresholds
Sigmoid scores aren't calibrated probabilities. The per-tag thresholds Project Sidewalk uses to suggest that a tag is
present or absent are in
tag_thresholds.json in
sidewalk-ai-api.
Evaluation
See the paper for per-tag results on the test split of
sidewalk-tagger-ai-validated.
REPRODUCE_RESULTS.md in
sidewalk-tagger-ai reproduces them.
Citation
@inproceedings{10.1145/3663548.3688531,
author = {Liu, Xinlei and Wu, Kevin and Kulkarni, Minchu and Saugstad, Michael and Rapo, Peyton Anton and Freiburger, Jeremy and Hosseini, Maryam and Li, Chu and Froehlich, Jon E.},
title = {Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset},
year = {2024},
publisher = {Association for Computing Machinery},
doi = {10.1145/3663548.3688531},
booktitle = {Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility},
articleno = {103},
numpages = {12}
}