DinoDepth — SDT depth heads on a frozen DINOv3 backbone
Trained Simple Depth Transformer (SDT) decoder heads for zero-shot affine-invariant (relative) monocular depth, reproducing AnyDepth (arXiv:2601.02760) on a frozen DINOv3 backbone. Only the small SDT decoder is trained; the DINOv3 encoder is frozen and loaded separately from Meta's checkpoints. Two heads are provided in this repo:
| File | Backbone | Decoder params | Train |
|---|---|---|---|
sdt-vitl16.safetensors |
DINOv3 ViT-L/16 | 13.4 M | 5 epochs |
sdt-vits16.safetensors |
DINOv3 ViT-S/16 | 5.5 M | 10 epochs |
These are decoder weights only (~13/5 M params) — pair each with its matching frozen DINOv3
backbone (facebook/dinov3-vitl16-pretrain-lvd1689m / -vits16-).
Zero-shot benchmark (our protocol)
AbsRel ↓ / δ1 ↑ on NYUv2 (Eigen 654) and KITTI (Eigen 652), scored with per-image least-squares scale+shift alignment in disparity space, Eigen/Garg crop, 10 m / 80 m cap.
| Model | NYU AbsRel | NYU δ1 | KITTI AbsRel | KITTI δ1 |
|---|---|---|---|---|
| ViT-L/16 + SDT (this repo) | 0.068 | 0.955 | 0.093 | 0.911 |
| ViT-S/16 + SDT (this repo) | 0.091 | 0.917 | 0.115 | 0.852 |
| AnyDepth ViT-L (paper) | 0.060 | — | 0.086 | — |
| AnyDepth ViT-S (paper) | 0.082 | — | 0.102 | — |
A faithful reproduction — ~0.01 AbsRel behind the paper on each benchmark (consistent across both backbones; plausibly the augmentation/data-filtering details AnyDepth underspecifies).
Usage
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from dinov3_depth.head import DepthModel, DepthModelConfig
# Frozen DINOv3 ViT-L/16 + (randomly-initialised) SDT head; default config matches the trained head
# (GroupNorm, fusion_channels=256).
model = DepthModel.from_pretrained(DepthModelConfig(backbone="vitl16"))
head = hf_hub_download("blanchon/dinodepth-model", "sdt-vitl16.safetensors")
model.head.load_state_dict(load_file(head))
model.eval()
# images: float [B, 3, H, W] in [0, 1], H and W multiples of 16. Returns affine-invariant disparity.
disparity = model(images)
(Use backbone="vits16" + sdt-vits16.safetensors for the small head.)
Architecture & training
- Encoder: frozen DINOv3 ViT (LVD-1689M), patch 16; 4 intermediate layers tapped.
- Decoder (SDT): softmax-fuse the 4 tapped layers at the patch grid → depthwise detail enhancer → two learned DySample ×4 upsamplers → output conv. GroupNorm, fusion width 256.
- Loss: scale-and-shift-invariant + multi-scale gradient matching (1:2), on disparity.
- Data: the harmonized 369K-image corpus at
blanchon/dinodepth-dataset(Hypersim, VKITTI2, BlendedMVS, IRS, TartanAir). 768² input, AdamW lr 1e-3, PolyLR.
References
- AnyDepth (arXiv:2601.02760) · DINOv3 (arXiv:2508.10104)
- Dataset: blanchon/dinodepth-dataset