ADE20K probe on DINOv3 ViT-S/16 at 144 px

A linear ADE20K semantic segmentation probe on frozen DINOv3 ViT-S/16 features at 144 px, a passive-vision reference for CanViT, trained with the paper's probing protocol.

CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.

Paper (NeurIPS 2026) · Code · Project page · All checkpoints

Usage

uv add "canvit-pytorch>=0.2" transformers

import torch
from PIL import Image
from canvit_pytorch.preprocess import preprocess
from canvit_pytorch.probes import SegmentationProbe
from canvit_pytorch.teacher import load_teacher

dinov3 = load_teacher("facebook/dinov3-vits16-pretrain-lvd1689m", torch.device("cpu"))
probe = SegmentationProbe.from_pretrained("canvit/probe-ade20k-40k-dv3s-144px").eval()
scene = preprocess(144)(Image.open("scene.jpg").convert("RGB")).unsqueeze(0)  # [1, 3, 144, 144]

with torch.inference_mode():
    patches = dinov3(scene).patches.unflatten(1, (9, 9))  # [1, 9, 9, 384]
    logits = probe(patches)  # [1, 150, 9, 9]

Details


Features	facebook/dinov3-vits16-pretrain-lvd1689m patch features of 144 px images
Probe	dropout, BatchNorm, 1 × 1 convolution
Training steps	40,000, batch size 16
Released weights	step 29,500, selected on ADE20K validation mIoU
Optimizer	AdamW, peak learning rate 0.0003, weight decay 0.001
Schedule	1,500-step linear warmup, then cosine decay
Augmentation	DINOv3's segmentation training transforms: scale jitter by a factor of 0.5 to 2, random 512 px crops with its class-ratio crop rejection, horizontal flips with probability 0.5, photometric distortion
Dropout	0.1
Precision	bfloat16 autocast

License

MIT. The probe reads features of DINOv3 ViT-S/16, which is released under the DINOv3 License.

Citation

@inproceedings{berreby2026canvit,
  title={CanViT: Toward Active-Vision Foundation Models},
  author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  eprint={2603.22570},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.22570}
}

canvit-pytorch 0.1

This repository's files for canvit-pytorch 0.1 remain at revision canvit-pytorch-0.1: with canvit-pytorch<0.2, pass revision="canvit-pytorch-0.1" to from_pretrained.