Project Sidewalk Tagger: Surface Problem

A multi-label classifier that predicts which accessibility tags apply to a Surface Problem label in a crop of a Google Street View image. Project Sidewalk serves it through sidewalk-ai-api to suggest tags for crowdsourced labels.

This is the DINOv2 model trained on the cleaned dataset in Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset (ASSETS '24). Training and evaluation code is in sidewalk-tagger-ai.

Provenance

validated-dino-cls-b-surfaceproblem-tags-best.pth is a byte-for-byte copy of the file of the same name in projectsidewalk/sidewalk-tagger-ai-models at commit 65959dbc80b87e4f39385204c4b639cbcf58e1a8, which holds every model from the paper and remains available unchanged.

sha256: be88b422572a783922fa13f9b63009d5753d7c17039928375fb2c1d0fd4aa335

Model

• Backbone: DINOv2 ViT-B/14 with 4 register tokens (dinov2_vitb14_reg4), fine-tuned.
• Head: Linear(768, 256) → ReLU → Linear(256, 13), with a sigmoid per class.
• Input: RGB crop resized to 256 px and padded to a multiple of 14, normalized with ImageNet mean and std.
• Format: a PyTorch state dict; it loads with torch.load(..., weights_only=True).

The architecture is custom rather than a transformers class. See DinoVisionTransformerClassifier and ImageTagger in sidewalk_ai_api/tagger.py for loading and preprocessing.

Classes

The output has 13 classes, in this order. NULL entries are unused placeholders that preserve the class indices from training.

Thresholds

Sigmoid scores aren't calibrated probabilities. The per-tag thresholds Project Sidewalk uses to suggest that a tag is present or absent are in tag_thresholds.json in sidewalk-ai-api.

Evaluation

See the paper for per-tag results on the test split of sidewalk-tagger-ai-validated. REPRODUCE_RESULTS.md in sidewalk-tagger-ai reproduces them.

Citation

@inproceedings{10.1145/3663548.3688531,
  author = {Liu, Xinlei and Wu, Kevin and Kulkarni, Minchu and Saugstad, Michael and Rapo, Peyton Anton and Freiburger, Jeremy and Hosseini, Maryam and Li, Chu and Froehlich, Jon E.},
  title = {Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset},
  year = {2024},
  publisher = {Association for Computing Machinery},
  doi = {10.1145/3663548.3688531},
  booktitle = {Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility},
  articleno = {103},
  numpages = {12}
}