CanViT-B fine-tuned on ImageNet-1k, 8 × 8 canvas (JAX / Flax NNX)
canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02 fine-tuned end to end for ImageNet-1k classification by linear probing then fine-tuning (LP-FT) with its canvas at 8 × 8, in the checkpoint format of canvit-nnx.
CanViT, the Canvas Vision Transformer, is an active-vision foundation model: it sees a scene through a sequence of glimpses and remembers it on a scene-wide canvas.
Paper (NeurIPS 2026) · Code · Project page · All checkpoints
Install
uv add canvit-nnx
Usage
import jax.numpy as jnp
from PIL import Image
from canvit_nnx import CanViTForImageClassification, Viewpoint, sample_at_viewpoint
from canvit_nnx.preprocess import preprocess
scene = jnp.asarray(preprocess(512)(Image.open("scene.jpg").convert("RGB")))[None]
model = CanViTForImageClassification.from_pretrained("canvit/canvitb16-add-vpe-finetune-g128px-s512px-c8-in1k-2026-08-01-nnx")
model.eval()
state = model.init_state(batch_size=1, canvas_grid_size=8)
viewpoint = Viewpoint.full_scene(batch_size=1)
glimpse = sample_at_viewpoint(spatial=scene, viewpoint=viewpoint, glimpse_size_px=128)
logits, state = model(glimpse=glimpse, state=state, viewpoint=viewpoint)
Results
Top-1 accuracy (%) on the ImageNet-1k validation set (C2F viewpoints, 512 px scenes, 8 × 8 canvas): the mean over 11 runs, with a 95% bootstrap interval where runs differ.
| Glimpses | Top-1 | 95% interval |
|---|---|---|
| 1 | 80.54 | — |
| 3 | 83.09 | 83.04 to 83.13 |
Details
| Backend | JAX / Flax NNX |
| Model class | CanViTForImageClassification |
| Converted from | canvit/canvitb16-add-vpe-finetune-g128px-s512px-c8-in1k-2026-08-01-nnx at 518c9fbd59c6ed3f8b8cfefc9416b7d66ac22c55 |
| Conversion checked at | 512 px scenes, 128 px glimpses, 8 × 8 canvas |
| Classes | 1000 |
| Fine-tuning | LP-FT from canvit/canvitb16-add-vpe-pretrain-g128px-s512px-in21k-dv3b16-2026-02-02, 100,080 steps, with JAX/Flax NNX on Cloud TPU |
| Evaluated | the canvit-pytorch 0.1 export at revision 518c9fbd, which this conversion reproduces (provenance.json) |
License
MIT. CanViT is pretrained by distillation from DINOv3, which is released under the DINOv3 License.
Citation
@inproceedings{berreby2026canvit,
title={CanViT: Toward Active-Vision Foundation Models},
author={Berreby, Yoha{\"i}-Eliel and Du, Sabrina and Durand, Audrey and Krishna, B. Suresh},
booktitle={Advances in Neural Information Processing Systems},
year={2026},
eprint={2603.22570},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2603.22570}
}