CanViT-B, pretrained on ImageNet-21k, fine-tuned on ImageNet-1k
CanViT-B, pretrained on ImageNet-21k, fine-tuned end to end for ImageNet-1k classification by linear probing then fine-tuning (LP-FT).
CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.
Paper (NeurIPS 2026) · Code · Project page · All checkpoints
Usage
uv add "canvit-pytorch>=0.2"
import torch
from PIL import Image
from canvit_pytorch import CanViTForImageClassification, Viewpoint, sample_at_viewpoint
from canvit_pytorch.benchmarks.imagenet import CLASS_NAMES
from canvit_pytorch.preprocess import preprocess
model = CanViTForImageClassification.from_pretrained("canvit/canvitb16-add-vpe-finetune-g128px-s512px-in1k-2026-07-24").eval()
scene = preprocess(512)(Image.open("scene.jpg").convert("RGB")).unsqueeze(0) # [1, 3, 512, 512]
state = model.init_state(batch_size=1, canvas_grid_size=32)
with torch.inference_mode():
viewpoint = Viewpoint.full_scene(batch_size=1, device=scene.device)
glimpse = sample_at_viewpoint(spatial=scene, viewpoint=viewpoint, glimpse_size_px=128)
logits, state = model(glimpse=glimpse, state=state, viewpoint=viewpoint) # [1, 1000]; more glimpses refine it
print(CLASS_NAMES[logits.argmax().item()])
Results
Top-1 accuracy (%) on the ImageNet-1k validation set (C2F viewpoints, 512 px scenes, 32 × 32 canvas): the mean over 11 runs, with a 95% bootstrap interval where runs differ.
| Glimpses | Top-1 | 95% interval |
|---|---|---|
| 21 | 84.44 | 84.42 to 84.46 |
Details
| Initialization | canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02, its CLS readout fused with the DINOv3 probe canvit/dinov3-vitb16-lvd1689m-in1k-512x512-linear-clf-probe |
| Rollouts | 4 glimpses of 128 px on 512 px scenes, 32 × 32 canvas |
| Training | 100,000 of 100,080 steps, batch size 256 |
| Optimizer | AdamW, learning rate 2.5e-05, weight decay 0.0001, gradient clipping 1 |
| Schedule | 25,000-step linear warmup, then cosine decay to 0 |
| Loss | cross-entropy after every glimpse, label smoothing 0.1 |
| Trainer | JAX/Flax NNX on Cloud TPU, exported to PyTorch |
License
MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.
Citation
@inproceedings{berreby2026canvit,
title={CanViT: Toward Active-Vision Foundation Models},
author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
booktitle={Advances in Neural Information Processing Systems},
year={2026},
eprint={2603.22570},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.22570}
}