CanViT classifier (MLX)

canvit/canvitb16-add-vpe-finetune-g128px-s512px-in1k-2026-04-06 in the checkpoint format of canvit-mlx, for MLX.

CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.

Paper (NeurIPS 2026) · Code · Project page · All checkpoints

Install

uv add canvit-mlx

Usage

import mlx.core as mx
from PIL import Image

from canvit_mlx import CanViTForImageClassification, Viewpoint, sample_at_viewpoint
from canvit_mlx.preprocess import preprocess

scene = mx.array(preprocess(512)(Image.open("scene.jpg").convert("RGB")))[None]
model = CanViTForImageClassification.from_pretrained("canvit/canvitb16-add-vpe-finetune-g128px-s512px-in1k-2026-04-06-mlx")
model.eval()
state = model.init_state(batch_size=1, canvas_grid_size=32)
viewpoint = Viewpoint.full_scene(batch_size=1)
glimpse = sample_at_viewpoint(spatial=scene, viewpoint=viewpoint, glimpse_size_px=128)
logits, state = model(glimpse=glimpse, state=state, viewpoint=viewpoint)

Details

License

MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.

Citation

@inproceedings{berreby2026canvit,
  title={CanViT: Toward Active-Vision Foundation Models},
  author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  eprint={2603.22570},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2603.22570}
}