Argus-Colqwen3.5-2b-v0 · bf16 release

Argus: Region-Aware Query-Conditioned Mixture of Experts for Visual Document Retrieval University of Innsbruck — Data Science group · 2026

DataScience-UIBK/Argus-Colqwen3.5-2b-v0-bf16 is the bfloat16 merged release of Argus-Colqwen3.5-2b. It is the exact same trained network as the fp32 sibling DataScience-UIBK/Argus-Colqwen3.5-2b-v0; only the LoRA-merge dtype differs.

The bf16 release is half the disk size (4.6 GB vs 9.3 GB), faster to download, and easier to deploy on memory-constrained GPUs.

At the 2B scale the merge-time bf16 cast lands inside the nDCG@5 measurement noise floor — every per-task score is bit-identical to the fp32 sibling (see the per-task table below). The bf16 release is therefore the strict winner for almost every deployment scenario at this size.

Use this bf16 release unless you specifically need fp32 for downstream merging / quantisation chains.

TL;DR — leaderboard standing

• Strong on the ViDoRe v1 leaderboard at the 2B scale (V1 = 0.9149) — competitive with nomic-ai/colnomic-embed-multimodal-3b (V1 = 0.916) at 2/3 the parameter count.
• Best 2B-class result on V2 (V2 = 0.6152), comfortably ahead of vidore/colpali-v1.3 and Metric-AI/colqwen2.5-3b-multilingual.
• 2.3 B parameters, 1024-d per-token embedding, ≤ 2048 visual tokens / page — fits on a single 8 GB GPU at bf16 inference.
• Apache 2.0, training pipeline trained on public ViDoRe + VDR-Multilingual subsets only.

What is novel here

Most ColPali-style retrievers project every visual token through the same dense head, no matter what the query is. Argus replaces that dense head with a sparse mixture in which the gates depend on both the visual token and a pooled query summary, so the same page gets routed differently for different queries:

1. Region pooling — visual tokens are grouped into 4-token regions before routing.
2. Query-conditioned latent gating (GateScalars) — router input is region + region_coord_proj(coords) + query_context_proj(pooled_query). The query summary makes routing task-aware: a financial-numbers query routes through a different expert than a layout query, on the same page.
3. Sparse top-k=2 of 4 latent specialists, fused with an always-on shared dense expert via two learnable gating scalars.
4. Region-aware load balancing — load balance + KL-uniform + 0.01·router-z² aux losses suppress routing collapse.
5. 3-stage curriculum — dense baseline (teacher) → MoE balance warmup → joint retrieval with KL distillation.

Router sits at backbone layer −5.

Model details

Why two dtypes (and why bf16 is essentially free at 2B)

Merging a LoRA into the base requires materialising (α/r)·A·B and adding it to the base weight matrix.

• In bf16, both the delta cast and the addition lose precision — the gap is small but irreversible.
• In fp32, both are exact.

For the 4B sibling, this merge cost shows up as ~0.1 pp on V1 and ~0.2 pp on V2. For the 2B model it does not — every per-task ViDoRe score is bit-identical to the fp32 sibling. We attribute this to the small 2B model not having "borderline" routing decisions that the bf16 cast can flip; the late-interaction MaxSim pooling further averages out the residual noise across many tokens.

So at the 2B scale the bf16 release is the strict winner: same scores, half the disk, faster I/O. We still publish fp32 for users who specifically need maximum-precision weights for downstream merging or quantisation.

Performance — ViDoRe v1 (English, nDCG@5, 10 tasks)

Per-task scores measured with mteb 2.12 on the published bf16 weights, side-by-side with the fp32 sibling and the 4B variants for transparency.

ViDoRe v1 — 2B / 3B-class leaderboard comparison

Argus matches nomic's 3B-class result at smaller scale and a wider per-token dim, and is the strongest sub-3B retriever published to date.

Performance — ViDoRe v2 (English, nDCG@5, 4 tasks)

ViDoRe v2 — 2B / 3B-class context

ViDoRe v3

Not yet evaluated for this release.

Storage cost

Per-page storage is identical between the 2B and 4B Argus releases — both use the same 1024-d head and same 2048-token visual budget. The choice between them is inference cost (2B is ~50 % faster than 4B), not corpus storage.

Installation

pip install "transformers>=5.0.0,<6.0.0"
pip install "mteb>=2.12,<3.0.0"
pip install -U "transformers>=5.0,<6.0"
pip install flash-attn==2.6.3 --no-build-isolation     # optional
rm -rf ~/.cache/huggingface/modules/transformers_modules

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor

MODEL_ID = "DataScience-UIBK/Argus-Colqwen3.5-2b-v0-bf16"
DEVICE   = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModel.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,                  # this release ships in bf16
    attn_implementation="flash_attention_2",
    device_map=DEVICE,
).eval()

processor = AutoProcessor.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    max_num_visual_tokens=2048,
)

queries = [
    "What is the company's revenue in 2019?",
    "How does the proposed model compare to baselines?",
]
documents = [
    Image.open("page_a.png").convert("RGB"),
    Image.open("page_b.png").convert("RGB"),
]

q_emb  = model.encode_queries(processor, queries)
d_emb  = model.encode_images(processor, documents)
scores = processor.score(q_emb, d_emb)
print(scores)

Reproduce ViDoRe results with MTEB

import mteb

m  = mteb.get_model("DataScience-UIBK/Argus-Colqwen3.5-2b-v0-bf16")
v1 = mteb.get_benchmark("ViDoRe(v1)").tasks
v2 = mteb.get_benchmark("ViDoRe(v2)").tasks
mteb.MTEB(tasks=v1 + v2).run(m, encode_kwargs={"batch_size": 4})

Training

Same recipe as the fp32 sibling — see its card for full details. Only difference is the merge dtype.

When to use 2B-bf16 vs the other variants

Limitations

• English-dominant.
• Merge-time bf16 cast is irreversible — you cannot recover fp32 numbers by upcasting after load. (At 2B this does not matter — the bf16 cast is numerically equivalent to fp32 on ViDoRe; it does matter at 4B.)
• ViDoRe v3 not yet evaluated.

License

Apache 2.0, inherited from Qwen3.5-VL-2B-Instruct.

Citation

@article{abdallah2026argus,
  title={Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval},
  author={Abdallah, Abdelrahman and Abdalla, Mahmoud and Ali, Mohammed and Jatowt, Adam},
  journal={arXiv preprint arXiv:2606.04300},
  year={2026}
}

Contact

• Org: DataScience-UIBK, University of Innsbruck
• Issues: open one on this repo's Community tab.