vllm-sr/vela3-mini

Name it. Vela finds it.
_{vLLM Semantic Router × KR Labs · Vela 3.0 family · vela3-mini · 4B · base d3-mini · Apache-2.0}

What it does · Quick start · Evaluation · The family · 📡 Collection

---

vela3-mini is the 4B member of the Vela 3.0 family, built on d3-mini. In one system_one call it answers routing and general decisions and finds the exact spans you name (entities, relations, personal data, unsupported claims, evidence), with probabilities and character offsets and no generated text.

Summary

vela3-mini is the balance point of the Vela 3.0 family. d3-mini, frozen, answers every ordinary decision, bit for bit, and gated adapters add the spans you name, entities, relations and router signals. At 4B it is at or above every model the GLiFormer paper reports on classification and relations, and it writes its own definition for each label you name.

Highlights

• Zero-shot classification: 80.2 on the 13-set GLiFormer classification suite, 0.4 above GPT-5-mini: 79.8 (reported, GLiFormer paper Tab 5).
• Relations: 30.1 end to end (GLiFormer Table 4 protocol, avg of 4), above every model reported (best: Qwen3.5-9B, prompted, 25.7; reported, GLiFormer paper Tab 4).
• Zero-shot NER: 65.7 on OOD-7 with its own label definitions (label names in; written at inference and cached per label set); 63.3 from label names alone.
• Evidence: 45.3 word-F1 on QASPER, zero-shot, against Vela 2.0 4B: 31.5 (same harness).
• Router signals: better than Vela 2.0 4B on 18 of the 24 test files of the public router signal suite, level on 5, worse on 1 (paired 95% CI).
• Decisions, bit for bit: every ordinary decision is answered by d3-mini@566da0ea itself, adapters off (561 of 561 identical in our check).

_{vela3-mini (4B) against reported numbers. Accent = vela3-mini; grey = other models (reported).}

At a glance

What it does

Quick start

# pip install "transformers==5.17.0" torch torchvision pillow opencv-python-headless safetensors accelerate
from transformers import AutoModel

m = AutoModel.from_pretrained("vllm-sr/vela3-mini", trust_remote_code=True)  # downloads d3-mini at the pinned revision

# Name any labels; the spans come back with offsets and scores.
r = m.system_one(
    state="Geoffrey Hinton presented capsule networks at NeurIPS in Montreal.",
    questions={"ents": {"type": "span", "criteria": ["researcher", "algorithm", "conference", "location"]}})
# researcher "Geoffrey Hinton", algorithm "capsule networks", conference "NeurIPS", location "Montreal"

# Unsupported claims in an answer, against a context or an image.
r = m.system_one(
    state={"context": "The Eiffel Tower is 330 metres tall and was completed in 1889.",
           "answer": "The Eiffel Tower is 330 metres tall and was completed in 1925 by Gustave Eiffel's son."},
    questions={"halu": {"type": "span", "criteria": {"unsupported": None}}})
# flags "1925 by Gustave Eiffel's son"; add images=["photo.jpg"] to check an answer against a photo

# PII, relations and evidence use the same call.
r = m.system_one(
    state="Steve Jobs founded Apple in Cupertino together with Steve Wozniak.",
    questions={"pii": {"type": "span", "criteria": ["PERSON"]},
               "rel": {"type": "span", "instructions": "Which spans are the founder of Apple?", "criteria": {"founder": None}}})

# Any other choice / yes-no / score question can opt in to the optional decision adapter: add "decision_adapter": True to it.

Evaluation

All numbers on public test sets, measured through this package with system_one, one request at a time and the shipped thresholds. Other models' numbers are as reported by their authors (or scored in the same harness where marked). Full per-set tables: EVALUATION.md.

Named entities

Label names are the only input; the model writes its own definitions at inference (d3-mini itself, adapters off) and caches them per label set. Training holds no CrossNER or MIT text, though many generic CrossNER type names occur as training labels.

Relations

CrossRE 22.5 (in-domain) is above every model reported in the GLiFormer paper (best: GLiNER-Relex 18.1).

Evidence

ACL-Verbatim is in-domain: its silver training split is in the training data.

Unsupported claims (hallucination spans) Zero-shot classification

These questions run on d3-mini with every adapter off.

Router signals (public router signal suite; above Vela 2.0 4B on 6 of 8)

vllm-sr/router-signal-suite (public), the test file of each signal, scored with the dataset's own scoring code. Each signal is a router question this package was trained on (supervised, as in Vela 2.0), asked one request at a time with the shipped thresholds.

Vela 2.0 4B was run by us through its public release on the same rows. Decision-1.0-Route-0.6B is scored from the suite's published predictions; it was trained on the suite's own train split, so these test files are in-distribution for it.

Bold: the paired 95% bootstrap CI against Vela 2.0 excludes a tie. The last column drops the test rows that share text (exact or a 13-gram) with this model's training data. Safety and jailbreak are scored on the rows whose text the suite publishes (2,230 of 3,003 and 1,878 of 3,000).

Domain and feedback are also trained on the suite's own train split (with the public router data Vela 2.0 used; the suite's test and hold rows, and anything sharing an exact text or a 13-gram with them, are kept out), as Decision-1.0-Route is, so their test files are in-distribution for both. The domain question has a 15th option, religion, which the suite maps to philosophy. Feedback has NO_FEEDBACK as a trained option.

Decisions (general decisions are the base model's, bit for bit)

General decisions are the base model's own answers, bit for bit, so the base's board entry is this package's decision score. Full score (the board's ranking): 60% private same-skill tests and 40% private new-domain tasks; public index: the 37 public benchmarks, chance-corrected. All values are board entries on the Decision Index 0.4.1 board (multimodalart/jev-decision-index, data generated 2026-10-28T09:02Z) unless noted; the rank is the position in the board's Full-score order, and tied rank marks entries the board shows as tied. On the board's Full score, d3-mini scores 49.97 (rank 87, tied). This package pins d3-mini revision 566da0ea; the board does not state which d3-mini revision it ran.

MASSIVE: three scenarios never used in our router training (iot, transport, cooking), 52 locales; accuracy on the 14,911 test utterances that match no training text of the model's lineage. The 342 excluded utterances occur verbatim in upstream training mixtures (tasksource MASSIVE / HWU64 rows). Vela 2.0 values are on the full 15,253, so these rows are not bolded. Super-NI 1568: 79 to 90 of its 162 ProPara passages share a 13-gram with other tasks' training rows.

Latency

Span questions with label names only. A new label set first gets its definitions written, once: 0.40 s for 4 labels, 0.37 s for 10 (median); then cached.

Benchmarks

• OOD-7 (CrossNER AI, literature, music, politics, science; MIT movie, restaurant): find the entities of the named types in text from domains the model was not trained on (strict entity F1).
• NER, 26 datasets of the GLiFormer paper (IE_INSTRUCTIONS): entities of named types across 26 public NER sets, not all zero-shot (strict entity F1).
• CoNLL04, head entity given: given one entity of a sentence, find the entities it is related to by the named relation (F1 over those tail entities).
• Relations end to end, GLiFormer Table 4 protocol (CoNLL04, DocRED, FewRel, CrossRE): find the entity pairs and their relation from raw text (strict triplet F1).
• QASPER: find the sentences of a research paper that answer a question about it (word-F1).
• ACL-Verbatim (KRLabsOrg/acl-verbatim-spans): find the passage of an ACL paper that supports an answer (word-F1).
• M-HalDetect (validation): which words of an image description the image does not support (char-F1).
• LettuceDetect v2 test set (10,698 answers, as in the LettuceDetect v2 paper (GroundLM 2026 workshop at EMNLP 2026) and card): does an answer contain claims its context does not support (example-F1: the answer is flagged or not).
• RAGTruth: does a retrieval-augmented answer contain claims its retrieved context does not support (example-F1).
• MASSIVE, three unseen scenarios (iot, transport, cooking; 52 languages): pick the intent and the scenario of a user command the model never saw in training (accuracy).
• Super-NI task 1568 (ProPara): answer a multiple-choice question about how a process changes an entity (accuracy).
• Zero-shot classification, 13 sets (the GLiFormer paper's suite: SST-2, IMDB, AG News, Banking77, and more): pick the label of a text from label names only (mean macro-F1).
• Decision Index 0.4.1 (board, 2026-10-28): 37 public decision benchmarks plus private tests; the Full score ranks on the private same-skill and new-domain tests (chance-corrected skill, 0 = guessing).
• Router signal suite (vllm-sr/router-signal-suite): the vLLM Semantic Router's public test of its request signals (harmful requests, jailbreaks, personal data, unsupported answers, fact-check need, modality, domain, feedback); each label comes from public corpora (AUC or accuracy).

The family

How it works

• Frozen base. d3-mini@566da0ea runs unchanged; ordinary questions are its own answers.
• Gated LoRA adapters. Each switches on only for its own questions, and every answer names the one it used (head_reason): decision (router signals), NER (entity labels with definitions), relation, image (hallucination spans with images), span (evidence, PII, unsupported claims, toxic spans, names-only labels), and an optional decision adapter for any other question ("decision_adapter": true).
• Spans as decisions. The labels are presented as options; a span head scores every word against every label, and agreeing neighbours form a span with character offsets.
• Self-written definitions. Before reading, the model (d3-mini itself, adapters off) writes a one-line definition for each undescribed label, cached per label set: OOD-7 goes from 63.3 (names) to 65.7.
• Specification. Adds 973 MB of adapters and heads; image threshold 0.55; text or JSON state, images and video for decisions, images for hallucination spans. Full evaluation: EVALUATION.md.

Trade-offs

• Unsupported claims: hallucination spans 0.874 against 0.921 for LettuceDetect v2 qwen-2b, RAGTruth 0.775 against 0.850 for HDM-2 (3B) (reported).
• General decisions are d3-mini's: its board entry is below StartLux-Decision v1.1 4B (full score 54.80 vs 49.97) and Perplexity Decider Flash v1.2 (4B) (full score 54.29 vs 49.97) on the Decision Index 0.4.1.
• NER from label names: 63.3 on OOD-7, level with the best reported encoder (gliner-multitask-v1.0 63.25) and below GNER-T5-xl (3B) 66.1 (reported).

Citation

@misc{vela3_mini,
  title        = {vela3-mini (Vela 3.0, 4B): zero-shot extraction and router signals on Decision 3.0},
  author       = {{KR Labs} and {vLLM Semantic Router}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/vela3-mini}}
}