ColUltraVBERT

ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from UltraVBERT. It encodes each query token and each page token as a 768-d vector and ranks pages by MaxSim. Truncating the vectors to 128, 256, 320 or 512 dimensions gives a smaller index.

Models

Data

Usage

Requires sentence-transformers>=6.1 and transformers>=5.21, which supports UltraVBERT natively (docs). On 5.20.x, pass trust_remote_code=True to load the code in this repository.

from PIL import Image
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("athrael-soju/ColUltraVBERT")

pages = [Image.open("page1.png"), Image.open("page2.png")]
queries = ["What was the 2023 revenue?", "Which table lists regional margins?"]

page_embeddings = model.encode_document(pages)
query_embeddings = model.encode_query(queries)

# One row per query, one column per page.
scores = model.similarity(query_embeddings, page_embeddings)

For a smaller index, keep the first 128, 256, 320 or 512 dimensions of each vector and L2-normalize it again.

Model

A linear layer projects each of UltraVBERT's 768-d token states to a 768-d retrieval vector. Vectors are L2-normalized and padding tokens are dropped. Pages are resized to at most 3,072 px on the longest edge and split into up to 25 tiles of 512 px, plus a downscaled view of the whole page: at most 1,664 image vectors per page, plus the prompt and tile-tag vectors.

Training

The released weights are a per-block merge of four fine-tunes of UltraVBERT on VDR Train 500k that differ in hard negatives, distillation and seed. Each fine-tune trains rank-32 LoRA adapters on the text encoder and the output projection for one epoch at batch size 256, with Matryoshka losses at 128, 256, 320, 512 and 768 dimensions; the vision encoder and connector stay frozen.

ViDoRe

ColUltraVBERT's scores are its official MTEB results (results#783): MTEB 2.26.3, 768-d vectors, bf16 and SDPA attention. The MTEB maintainers scored the private ViDoRe V3 tasks, Nuclear and Telecom (results#792). ColModernVBERT's V1 and V2 scores are our MTEB 2.26.3 runs of ModernVBERT/colmodernvbert at revision 9052fc0e; its V3 scores are its MTEB results.

Smaller indexes

Scores come from cached page features; at 768 dimensions and 3,072 px they are within 0.1 of the official means. For 2,048 px pages with 16 tiles, set size.longest_edge to 2048 and max_image_tiles to 16 in processor_config.json; a page then produces at most 1,088 image vectors.

Limits

• A page produces up to 1,664 image vectors, 2.4 MiB at 768 dimensions in bf16, plus its prompt and tile-tag vectors.

License

MIT. The UltraBERT-150M and SigLIP2-B/16-512 components are Apache-2.0, with their license text in LICENSE-APACHE.