Learned viewing policy for CanViT-B on ADE20K: REINFORCE, seed 0

A learned viewing policy: it scores candidate viewpoints from the canvas of the frozen CanViT-B and the segmentation its ADE20K probe reads out of the canvas, and CanViT-B looks next at the candidate of highest score. Trained by REINFORCE to choose the viewpoints whose glimpses most reduce the probe's cross-entropy.

CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.

Paper (NeurIPS 2026) · Code · Project page · All checkpoints

Usage

uv add "canvit-pytorch>=0.3"

import torch
from PIL import Image
from canvit_pytorch.preprocess import preprocess
from canvit_pytorch.rl import LearnedPolicy

policy = LearnedPolicy.from_pretrained("canvit/pgpolicy-ade20k-c64-t5-floorband-2026-07-07-s0", torch.device("cpu"))  # with the CanViT and probe it was trained with
scene = preprocess(512)(Image.open("scene.jpg").convert("RGB")).unsqueeze(0)  # [1, 3, 512, 512]

with torch.inference_mode():
    steps = policy.rollout(scene, num_glimpses=5)  # the full scene, then the policy's choices
for step in steps:
    print(step.t, step.viewpoint.centers.tolist(), step.viewpoint.scales.tolist())  # (row, col) center and scale
labels = policy.model.logits(steps[-1].state.canvas).argmax(dim=1)  # [1, 64, 64] ADE20K classes

Details


Observer	canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02 and its ADE20K probe canvit/probe-ade20k-40k-s512-c64-in21k, frozen
Viewing	512 px scenes, 128 px glimpses, a 64 × 64 canvas
Candidates	16 × 16 centers at each scale of 0.5, 0.25
Objective	REINFORCE
Training	5-glimpse episodes on the ADE20K training scenes, seed 0; the full record is under `training_run/`, as the original training code wrote it
Selected step	7,000, of lowest mean ADE20K validation cross-entropy after the glimpses the policy chose
Validation cross-entropy after each glimpse	0.7649, 0.7135, 0.6894, 0.6752, 0.6647

License

MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.

Citation

@inproceedings{berreby2026canvit,
  title={CanViT: Toward Active-Vision Foundation Models},
  author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  eprint={2603.22570},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.22570}
}

Original format

The files in the format of the code that trained this policy remain at revision original-format; canvit-pytorch reads those of main.