canvit/probe-ade20k-40k-s512-c64-in21k in the checkpoint format of canvit-nnx: an ADE20K segmentation probe for CanViT's canvas, loaded alongside the separately published CanViT checkpoint.
CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.
Paper (NeurIPS 2026) · Code · Project page · All checkpoints
uv add canvit-nnx
import jax.numpy as jnp
from PIL import Image
from canvit_nnx import CanViTForSemanticSegmentation, Viewpoint, sample_at_viewpoint
from canvit_nnx.preprocess import preprocess
scene = jnp.asarray(preprocess(512)(Image.open("scene.jpg").convert("RGB")))[None]
model = CanViTForSemanticSegmentation.from_pretrained_with_probe(
pretrained_repo="canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02-nnx",
probe_repo="canvit/probe-ade20k-40k-s512-c64-in21k-nnx",
)
model.eval()
state = model.init_state(batch_size=1, canvas_grid_size=64)
viewpoint = Viewpoint.full_scene(batch_size=1)
glimpse = sample_at_viewpoint(spatial=scene, viewpoint=viewpoint, glimpse_size_px=128)
logits, state = model(glimpse=glimpse, state=state, viewpoint=viewpoint) # [1, 64, 64, 150]
| Backend | JAX / Flax NNX |
| Model class | SegmentationProbe |
| Converted from | canvit/probe-ade20k-40k-s512-c64-in21k at b28ba9b866d3763cbea60035ef8b7b02eba5b01d |
| CanViT checkpoint | canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02-nnx |
| Classes | 150 |
| Example geometry | 512 px scenes, 128 px glimpses, 64 × 64 canvas |
MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.
@inproceedings{berreby2026canvit,
title={CanViT: Toward Active-Vision Foundation Models},
author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
booktitle={Advances in Neural Information Processing Systems},
year={2026},
eprint={2603.22570},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.22570}
}