VelaVec-T2I

Text→Image retrieval variant of VelaVec — a lightweight hybrid retrieval encoder (frozen distilled table + 3-layer bidirectional attention + dual pooling heads). This variant adds a vision projection head that maps CLIP teacher image features into the VelaVec text-embedding space, enabling cross-modal retrieval on top of the 9.8M-parameter text trunk.

Dedicated to the text→image direction (best single-head t2i on flickr30k eval). For image→text, see VelaVec-MultiModal (i2t r@1 = 0.650).

Model details

• Text trunk: identical to VelaVec — frozen 30,522×256 static table → 3 BiBlocks (RMSNorm, RoPE, SwiGLU) → attention pooling, dual heads (head=0 symmetric, head=1 retrieval).
• Vision head (config.vision): projection-only. Frozen CLIP teacher features — openai/clip-vit-base-patch32 (512d) + openai/clip-vit-large-patch14 (768d), L2-normalized and concatenated (1280d) — then a residual MLP (1280 → R768 → 256) into VelaVec space.
• No CLIP weights are bundled: at inference, teacher features are produced by the HF transformers CLIPModel (auto-downloaded from the configured repo ids, offline cache supported); only the small projection head (±1.5M params) is in model.safetensors.
• Projection training: all-5-description InfoNCE (caption→image softmax) with cosine-annealing + residual-anchor init from the best linear projection. No regression terms (they hurt t2i discrimination across images). Training data: flickr30k.

Usage

from modeling_velavec import VelaVec

model = VelaVec.from_pretrained("xagent2025/VelaVec-T2I")

# Text: VelaVec head-1 embeddings (retrieval head)
caps = model.encode(["a man riding a bicycle down a street"], head=1)

# Image: CLIP teacher features → projection head
from PIL import Image
imgs = [Image.open("photo.jpg")]
z_img = model.encode_image(imgs)          # [B,256] in the same space as text

score = z_img @ caps.T                     # caption→image ranking

Image preprocessing: resized/center-cropped to 224×224, CLIP normalization (same as teacher). No instruction prompting.

Files

Relationship to other models

• VelaVec — pure-text (no vision).
• VelaVec-T2I — text→image optimized (this repo).
• VelaVec-MultiModal — image→text balanced head (w-mse regression, i2t 0.650 / t2i 0.532).

Limitations

• Dedicated to t2i; its i2t (0.619) trails the balanced variant.
• Image inference requires PyTorch + transformers (teacher models); the pure-Rust path covers text only.
• Rust engine (text path, 53.9× vs teacher, cargo build, MIT): xagent2025/velavec-rust-inference
• flickr30k eval is a small paired retrieval pool (1000 images × 5000 captions) — indicative, not a full multimodal benchmark (e.g., COCO/Flickr30k 1K test protocols).