Loading…
Loading…
A learned viewing policy: it scores candidate viewpoints from the canvas of the frozen CanViT-B and the segmentation its ADE20K probe reads out of the canvas, and CanViT-B looks next at the candidate of highest score. Trained by Q-learning to choose the viewpoints whose glimpses most reduce the probe's cross-entropy.
CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.
Paper (NeurIPS 2026) · Code · Project page · All checkpoints
uv add "canvit-pytorch>=0.3"
import torch
from PIL import Image
from canvit_pytorch.preprocess import preprocess
from canvit_pytorch.rl import LearnedPolicy
policy = LearnedPolicy.from_pretrained("canvit/qpolicy-ade20k-c64-t5-2026-07-03", torch.device("cpu")) # with the CanViT and probe it was trained with
scene = preprocess(512)(Image.open("scene.jpg").convert("RGB")).unsqueeze(0) # [1, 3, 512, 512]
with torch.inference_mode():
steps = policy.rollout(scene, num_glimpses=5) # the full scene, then the policy's choices
for step in steps:
print(step.t, step.viewpoint.centers.tolist(), step.viewpoint.scales.tolist()) # (row, col) center and scale
labels = policy.model.logits(steps[-1].state.canvas).argmax(dim=1) # [1, 64, 64] ADE20K classes
| Observer | canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02 and its ADE20K probe canvit/probe-ade20k-40k-s512-c64-in21k, frozen |
| Viewing | 512 px scenes, 128 px glimpses, a 64 × 64 canvas |
| Candidates | 16 × 16 centers at each scale of 0.5, 0.25 |
| Objective | Q-learning |
| Training | 5-glimpse episodes on the ADE20K training scenes, seed 0; the full record is under training_run/, as the original training code wrote it |
| Selected step | 6,000, of lowest mean ADE20K validation cross-entropy after the glimpses the policy chose |
| Validation cross-entropy after each glimpse | 0.7649, 0.7143, 0.6885, 0.6741, 0.6657 |
MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.
@inproceedings{berreby2026canvit,
title={CanViT: Toward Active-Vision Foundation Models},
author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
booktitle={Advances in Neural Information Processing Systems},
year={2026},
eprint={2603.22570},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.22570}
}
The files in the format of the code that trained this policy remain at revision original-format;
canvit-pytorch reads those of main.