Model card for vit_giant_patch16_lingbot.robbyant

A LingBot-Vision ViT-Giant/16 image feature encoder. Pretrained with masked boundary modeling by the paper authors and converted to timm's Eva/DINOv3 implementation.

Model Notes

• Token layout: CLS at index 0, four register tokens at indices 1-4, patch tokens thereafter. The pretrained cfg uses global_pool='avg' over patch tokens; pass global_pool='token' at creation to reproduce the upstream CLS representation.
• fp32 forward outputs match the reference implementation with max abs diff 0.0e+00 on CLS, register, and patch tokens at 512x512 and 384x512. Conversion provenance, the pinned source revision, and per-partition parity metrics are recorded in manifest.json.
• Converted from https://huggingface.co/robbyant/lingbot-vision-vit-giant at revision f87d0865a0ae. The vit_giant_patch16_lingbot.robbyant architecture is pending in timm (PR); its pretrained cfg resolves the weights from this repo, so the usage below works on a timm checkout that includes the LingBot entrypoints.

Model Details

• Model Type: Image Feature Encoder
• Model Stats:
  • Params (M): 1134.5
  • GMACs: 1167.15
  • Activations (M): 654.86
  • Image size: 512 x 512
• Original: https://github.com/robbyant/lingbot-vision
• License: Apache 2.0
• Pretrain Dataset: 161M-image curated web corpus (see paper)
• Papers:
  • Vision Pretraining for Dense Spatial Perception: https://arxiv.org/abs/2607.05247
  • PyTorch Image Models: https://github.com/huggingface/pytorch-image-models

Model Usage

Image Classification

from urllib.request import urlopen
from PIL import Image
import timm
import torch

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model('vit_giant_patch16_lingbot.robbyant', pretrained=True)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)

Feature Map Extraction

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_giant_patch16_lingbot.robbyant',
    pretrained=True,
    features_only=True,
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # unsqueeze single image into batch of 1

for o in output:
    # print shape of each feature map in output
    print(o.shape)

Image Embeddings

from urllib.request import urlopen
from PIL import Image
import timm

img = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))

model = timm.create_model(
    'vit_giant_patch16_lingbot.robbyant',
    pretrained=True,
    num_classes=0,  # remove classifier nn.Linear
)
model = model.eval()

# get model specific transforms (normalization, resize)
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

output = model(transforms(img).unsqueeze(0))  # output is (batch_size, num_features) shaped tensor

# or equivalently (without needing to set num_classes=0)
output = model.forward_features(transforms(img).unsqueeze(0))
# output is unpooled, a (1, 1029, 1536) shaped tensor

output = model.forward_head(output, pre_logits=True)
# output is a (1, num_features) shaped tensor

Model Comparison

Explore the dataset and runtime metrics of this model in timm model results.

Citation

@article{lingbot-vision2026,
  title={Vision Pretraining for Dense Spatial Perception},
  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  journal={arXiv preprint arXiv:2607.05247},
  year={2026}
}

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/huggingface/pytorch-image-models}}
}