vllm-sr/vela3-fast-vision

Name it. Vela finds it.
_{vLLM Semantic Router × KR Labs · Vela 3.0 family · vela3-fast-vision · 0.3B · base UltraVBERT · MIT · preview}

What it does · Quick start · Evaluation · The family · 📡 Collection

---

vela3-fast-vision is the small vision member of the Vela 3.0 family, a preview: three 254M encoder backbones that answer decisions about images and text, mark the words of an answer that an image does not support, check OCR extractions against the scan, point at named things in an image, and answer the router's text signals, at about 30–40 ms per question on a GPU.

Summary

vela3-fast-vision is the image model of the Vela 3.0 family, at 0.3B parameters and small enough for a CPU. Three UltraVBERT backbones each answer their own kind of question: a decision backbone distilled from the d3 models for image and text decisions, a second decision-trained backbone for region pointing, and the original backbone, frozen, with adapters for image-hallucination spans, the OCR check and the router's text signals. One backbone pass per question. A preview.

Highlights

• Decisions: 37.5 on the Decision Index 0.3 public index and 22.79 on the Decision Index vision board 0.4, public part (our runs of the unmodified kits; the text result is also a board entry, Decision Index 0.4.1, with a Full score of 22.11).
• Region pointing: 0.441 mean IoU on BoxSet-test v2, level with a d3-turbo 27B dev build (0.442) at 1/89 the size.
• OCR check: char-F1 0.868 on a synthetic test set of 800 rendered documents (synthetic, not a public benchmark): 504 of 632 corrupted fields caught, 34 false flags on 11,981 clean fields.
• Image-grounded hallucination spans: 0.518 char-F1 on M-HalDetect validation (evaluation only).
• Text in the same package: unsupported claims 0.892 example-F1 on the 10,698-answer LettuceDetect v2 test set (LettuceDetect v2 mmBERT-base 0.869, reported); router signals better than Vela 2.0 0.3B on 12 of 24 test files, level on 9, worse on 3.
• Small and fast: one 0.3B backbone per question, 29–31 ms per question on one GPU at 1024 px.

At a glance

What it does

Quick start

# pip install "transformers>=5.20" torch safetensors tokenizers numpy scipy pillow huggingface_hub
import sys
from huggingface_hub import snapshot_download

path = snapshot_download("vllm-sr/vela3-fast-vision")
sys.path.insert(0, path)
from vela3v_encoder import Vela3VEncoder

m = Vela3VEncoder.load(path)  # GPU (bf16) if available, else CPU (fp32)

# 1. Image decisions: choice and yes/no questions (up to 3 images)
r = m.system_one("", {
    "kind": {"type": "choice", "instructions": "What type of document is this?",
             "criteria": {"invoice": "an invoice or bill", "letter": "a letter", "form": "a form"}},
    "total": {"type": "noul", "instructions": "Does this document show a total amount?"},
}, images=["scan.png"])
# r["answers"]["kind"] = {"type": "choice", "choice": ..., "probabilities": {...}}; r["answers"]["total"] = {"type": "noul", "noul": p(yes)}

# 2. Hallucination spans: the words of the answer that the image does not support (one image)
r = m.system_one({"user": "Describe the photo.", "answer": "Two dogs run on a beach at sunset."},
                 {"halu": {"type": "span", "criteria": {"unsupported": None}}}, images=["photo.jpg"])
# r["answers"]["halu"]["spans"]: the unsupported words, with character offsets

# 3. Region pointing: a coarse box and a point for a named thing (one image)
r = m.system_one("", {"where_total": {"type": "box", "instructions": "the total amount"}}, images=["scan.png"])
# r["answers"]["where_total"] = {"type": "box", "box": {"top", "bottom", "left", "right"}, "point": {"y", "x"}, "confidence"}

Images can be paths, PIL images, URLs or data URLs. A question the model cannot answer gets {"type", "error", "message"}; the other questions are still answered. Nothing is cut: a question longer than 32,768 tokens, or a decision with more than 3 images, gets an error instead.

Over HTTP (/v1/systemone, request body {"state", "questions", "images": [base64 data URLs]}):

python vela3_server.py --model vllm-sr/vela3-fast-vision --port 8000

Decision Index (text board: decision_index_engine:VelaVisionEngine; vision board: vela3v_di_engine:Vela3VEngine):

hf download vllm-sr/vela3-fast-vision --local-dir model
pip install -r model/requirements.txt "decision-index @ git+https://github.com/apolinario/decision-index@25c6ec7"
PYTHONPATH=model python -m decision_index run --engine decision_index_engine:VelaVisionEngine --option run=model --edition 0.3 --out runs/vela3-fast-vision --compact

Evaluation

All numbers on the full public test sets, measured through this package. No model of its size reports these sets, so the references are larger models. Details: EVALUATION.md.

Region pointing

BoxSet-test v2 is the box test set d3-turbo reports on: used for comparison only, never for training or for choosing settings.

Decisions on the Decision Index

Full score (the text board's ranking): 60% private same-skill tests and 40% private new-domain tasks, which only the board maintainers run; public index: the 37 public benchmarks, chance-corrected. The text row is the board entry (multimodalart/jev-decision-index, data generated 2026-10-28T09:02Z); its public index equals our own run of the kit on the public split. The rank is the position in the board's Full-score order, and tied rank marks entries the board shows as tied. The vision row is our run of the vision kit on its public split; Winoground (gated) was not run. Per-area and per-benchmark numbers are in EVALUATION.md.

OCR check (new)

Give the scan as the image and the OCR text or extracted fields as the state; the model marks the characters the scan does not support (a span question with "task": "ocr_check"). It is trained on synthetic rendered invoices, receipts, order forms and bank details only. On a synthetic test set of 800 rendered documents (synthetic, not a public benchmark) it reaches char-F1 0.868, catches 504 of 632 corrupted fields and flags 34 of 11,981 clean fields. Treat it as a preview.

Image decisions

Decisions come from the decision-trained backbone, which learned from d3 soft labels, including d3-mini's answers on RVL-CDIP train-split images; the test images are held out.

Image-grounded hallucination spans

M-HalDetect (CC BY-NC) was used for internal evaluation only; nothing was trained on it.

Text questions (the vela3-fast text adapters, retrained on this model's text encoder)

Other models' numbers as reported (or our run in the same harness where marked in the vela3-fast card); Vela 2.0 0.3B as published on its card. The hallucination test, RAGTruth and ACL-Verbatim have training splits in the span adapter's mix.

Router signals (public router signal suite; above Vela 2.0 0.3B on 6 of 8)

vllm-sr/router-signal-suite (public), the test file of each signal, scored with the dataset's own scoring code. Each signal is a router question this package was trained on (supervised, as in Vela 2.0), asked one request at a time with the shipped thresholds.

Vela 2.0 0.3B was run by us through its public release on the same rows. Decision-1.0-Route-0.6B is scored from the suite's published predictions; it was trained on the suite's own train split, so these test files are in-distribution for it.

Bold: the paired 95% bootstrap CI against Vela 2.0 excludes a tie. The last column drops the test rows that share text (exact or a 13-gram) with this model's training data. Safety and jailbreak are scored on the rows whose text the suite publishes (2,230 of 3,003 and 1,878 of 3,000).

Domain and feedback are also trained on the suite's own train split (with the public router data Vela 2.0 used; the suite's test and hold rows, and anything sharing an exact text or a 13-gram with them, are kept out), as Decision-1.0-Route is, so their test files are in-distribution for both. The domain question has a 15th option, religion, which the suite maps to philosophy. Feedback has NO_FEEDBACK as a trained option.

Over all 24 test files of the suite, including hazard and tool need (answered by the decision backbone, not by a trained router question): 12 wins, 9 ties and 3 losses against Vela 2.0 0.3B (paired 95% CI).

Latency

Each question is one encoder pass over the image and the question. The shipped adapters read images at 1024 px (vela3v_config.json).

Benchmarks

• BoxSet-test v2 (the box test set d3-turbo reports on; 5,512 targets + 749 absent): point at a named thing in an image (mean IoU of the box; point-in-box: the point falls inside the true box).
• RVL-CDIP (test subset, 2,000 scanned documents, 16 types): name the type of a scanned document (accuracy).
• M-HalDetect (validation, 786 image descriptions): which words of an image description the image does not support (char-F1).
• Decision Index 0.3 (board): a public suite of 37 decision benchmarks (chance-corrected skill, 0 = guessing; here the public part, our run).
• Decision Index vision board 0.4: a public suite of 11 image decision benchmarks (CV-Bench, BLINK, RealWorldQA, CharXiv, InfographicVQA, Mind2Web, Winoground, KIE, Hateful Memes, R-Bench-M, MMMU-Pro vision; chance-corrected skill; here the public part, our run, without the gated Winoground).

The family

How it works

• Three backbones. All are UltraVBERT (254M; revision 014dd536): SigLIP2 encodes each 512 px tile into patches, a connector merges them into 64 tokens per tile, and UltraBERT attends over image and text tokens together. The decision backbone (backbone_h2.safetensors) is fully trained as a decision model on d3 soft labels (text, images and d3-mini's answers on document images), on public datasets' train splits written as decisions and on synthetic decision tasks in the styles the Decision Index covers, labelled by d3 or by program and checked against every Decision Index item; three training runs were averaged. It answers every non-router choice and yes / no question, text or image, with its decision head. The region-pointing adapter runs on an earlier decision-trained backbone (backbone_v2.safetensors). The original backbone (model.safetensors, and its text encoder under text/) stays frozen and carries the adapters for spans, the OCR check and the router signals.
• Adapters. LoRA adapters, each with its own head: image-hallucination spans (the unsupported label over the answer's words), the OCR check, region pointing (a per-cell heat map read as a box and a point), and on the text side the router, span and NER adapters plus the domain and feedback task adapters.
• Training. Soft targets and box data from Decision 3.0 (d3, d3-turbo), shared by the Decision 3.0 team; image-hallucination pools built on COCO images; synthetic documents for the OCR check. Every threshold and checkpoint was chosen on dev data.

Trade-offs

• Hallucination spans are below the larger Vela 3.0 sizes: 0.518 against 0.575 for vela3-mini on M-HalDetect. Use vela3-mini or vela3-flash where span quality matters most.
• Decisions have no reasoning skills: document type is 0.713 against d3-mini's zero-shot 0.752; chart, STEM and GUI questions are weak. Region answers are coarse (64 × 64 px cells of a 512 px tile), so small targets such as a date cover about one cell.
• Scope: English only; one image per request for span and box questions, up to three for decisions (more is an error, never dropped). A preview: no claim of state-of-the-art results.

Licence

MIT for this model's adapters, code and card. The base athrael-soju/UltraVBERT is MIT; its components dmis-lab/UltraBERT-150M and google/siglip2-base-patch16-512 are Apache-2.0 (LICENSE, LICENSE-APACHE).

Citation

@misc{vela3_fast_vision,
  title        = {vela3-fast-vision (Vela 3.0, 0.3B, preview): image decisions, image-grounded hallucination spans and region pointing},
  author       = {{KR Labs} and {vLLM Semantic Router}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/vela3-fast-vision}}
}