Loading…
Loading…
The CanViT-B arm of the paper's architecture control, which compares CanViT-B with an AdaGlimpse baseline under the same DINOv3 teacher, data and glimpse budget: CanViT-B pretrained from a random initialization on ImageNet-1k images by dense latent distillation from DINOv3 ViT-B/16. The paper's flagship CanViT-B is a different checkpoint.
CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.
Paper (NeurIPS 2026) · Code · Project page · All checkpoints
uv add canvit-nnx
import jax.numpy as jnp
from PIL import Image
from canvit_nnx import CanViTForPretraining, Viewpoint, sample_at_viewpoint
from canvit_nnx.preprocess import preprocess
scene = jnp.asarray(preprocess(512)(Image.open("scene.jpg").convert("RGB")))[None]
model = CanViTForPretraining.from_pretrained("canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in1k-dv3b16-archctrl-2026-07-26-nnx")
model.eval()
state = model.init_state(batch_size=1, canvas_grid_size=32)
viewpoint = Viewpoint.full_scene(batch_size=1)
glimpse = sample_at_viewpoint(spatial=scene, viewpoint=viewpoint, glimpse_size_px=128)
output = model(glimpse=glimpse, state=state, viewpoint=viewpoint)
state = output.state
The paper reports this checkpoint through its ADE20K probe, canvit/probe-ade20k-40k-s512-c32-archctrl-nnx, whose card gives the mIoU after each glimpse, and through the COCO detector fine-tuned from it, canvit/canvitb16-add-vpe-detect-g128px-s512px-coco-from-archctrl-2026-07-27-nnx (appendices "Architecture control" and "COCO object detection").
| Backend | JAX / Flax NNX |
| Model class | CanViTForPretraining |
| Teacher | DINOv3 ViT-B/16 patch and CLS features of the scenes, precomputed and standardized by the run's per-position teacher moments, which the standardizers hold |
| Rollouts | 4 glimpses of 128 px on 512 px scenes, 32 × 32 canvas, backpropagation through all of them |
| Training | 26,875 steps at batch size 512, 2,500-step warmup, CLS loss weight 1, with JAX/Flax NNX on a Cloud TPU v6e-8 slice |
| Conversion | from the JAX checkpoint of the run, checked against the code that trained it on 8 ADE20K validation scenes over 4 glimpses: canvas relative L2 error at most 5.5e-03 in float32, where the original's attention kernels still compute in bfloat16; at least 99.85% of cells keep the probe's class (provenance.json) |
MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.
@inproceedings{berreby2026canvit,
title={CanViT: Toward Active-Vision Foundation Models},
author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
booktitle={Advances in Neural Information Processing Systems},
year={2026},
eprint={2603.22570},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.22570}
}