primitive-ai/primitive-router-lite-80m-onnx

Stop paying frontier prices for prompts that don't need them.

Send a chat messages array and one preference value.
Get back a ranked list of the LLMs worth calling — in about 190 ms, on a CPU you already own.

---

Why teams put a router in front of their LLM calls

Most production traffic is not hard. A sizable share of prompts are handled just as well by a mid-tier model as by the most expensive one in your account — but you can't tell which is which until you've read the prompt. So most stacks pin one model and overpay on the easy half of their traffic.

This model reads the prompt and ranks a 30-model roster for you.

• 💸 Cut spend up to 79.4%. The Economy preset averages $0.66 per million tokens against Balanced's $3.18 — for a 6.9% capability give-up.
• ⚡ No GPU, no PyTorch. Three pip packages, ~190 ms p50 on a shared 12-core box. It runs on the API server you already have.
• 🎚️ One dial, not a rules engine. economy / balanced / performance, or a raw alpha float if you want to tune it yourself.
• 🔒 Your prompts never leave your box. No API call, no telemetry, no service to authenticate against — the model reads the prompt locally and returns a ranking. Air-gapped deployments work.
• 🪜 Preset ordering that actually holds. route_ladder() guarantees Economy ≤ Balanced ≤ Performance per request, not just on average. Details below — this is the part most routers get wrong.
• 📊 Measured, not asserted. Every number on this page comes from 11,058 held-out test rows and is reproducible from presets.json.

---

Install & route in 60 seconds

pip install onnxruntime tokenizers numpy

from onnx_router import OnnxRouter

r = OnnxRouter("path/to/this/bundle")

messages = [
    {"role": "system", "content": "You are an expert mathematician."},
    {"role": "user", "content": "Prove that for every prime p > 3, p^2 - 1 is divisible by 24."},
]

out = r.route(messages, preference="balanced")   # "economy" | "balanced" | "performance"
# out = r.route(messages, alpha=0.37)            # or the raw slider, 0.0 .. 0.875

out["ranking"][0]["slug"]   # -> the model to call

That's the whole integration. No torch, no transformers, no service to stand up.

---

The numbers

Measured on 11,058 held-out test rows. Cost is real USD per million tokens at public vendor pricing. Capability and speed are given only as change relative to Balanced — absolute values come from a third-party index and are not published.

Guardrails, enforced at build time: capability drop never exceeds 18.8% and cost never exceeds 2.5× Balanced. A bundle that violates either does not ship.

alpha=0.5 is the neutral default, so ignoring the parameter reproduces no-preference behaviour. alpha is clamped to 0.875, the range the model was trained and verified over.

---

Router Pro — cheaper picks and better ones

The usual way to spend less on inference is to accept a worse answer. Measured on the same rows, Pro does not make you choose: at the neutral balanced setting it finds models that are 29.9% cheaper and 7.7% more capable than Lite picks.

That is not a bigger roster alone — it is a 350M encoder that reads the prompt more accurately feeding a 98-model roster. A router only saves what it can correctly identify, so the gain comes from all three parts at once: what kind of request this is (macro-F1 0.9031 vs 0.8825), how hard it is (tier accuracy 0.5355 vs 0.4749), and how many options it has to satisfy it (98 vs 30).

• 🔄 The roster keeps up with the market. New models are profiled and folded into Pro on a recurring catalog refresh, so the candidate list tracks what has actually shipped rather than what was current at training time — 178 models profiled, 98 routable today and growing. Lite's 30 are fixed until we cut a new bundle.

Same response shape, same one-line swap. You change a config value, not an integration.

A larger roster simply contains better value to find, and a stronger encoder is better at finding it — which is why the two gains compound instead of trading off.

Two honest notes on that table. The nDCG row compares both models on the identical 30-model roster, because ranking 98 candidates is a strictly harder problem than ranking 30 and the raw figures are not comparable — on its own 98-model roster Pro scores 0.9753, and quoting that against Lite's 0.9784 would understate it. And Lite genuinely wins on deployment: three pip packages, no GPU, no service to run. If you have no accelerator budget, stay on Lite.

Talk to us

Pro is available to enterprise organisations as a private deployment, or as a hosted inference endpoint if you would rather not run it yourself.

---

Independent benchmark: LLMRouterBench

Evaluated on LLMRouterBench (ACL 2026 Findings, arXiv 2601.07206) — 33 models, 21+ datasets, 400K+ instances, 10 routing baselines — using the benchmark's own prompt order, candidate pool, cached score/cost matrices and canonical prompt-level split, all unmodified.

Performance track — AvgAcc, macro-averaged over datasets

20-model candidate pool, 15 datasets, 2,297 held-out test rows, 5 seeds (42/1/7/123/2024). Lite lands +4.9 points above the best single model and level with EmbedLLM — from an 80M encoder on a CPU.

Performance-Cost track — accuracy per dollar

13 flagship models, 2,493 held-out test rows. Lite matches GPT-5's accuracy at $0.0309 per query against GPT-5's own $0.0609 — half the cost for the same answer quality. That is the highest CostSave of any method on this track, and it leads on PerfGain too.

We do not report ParetoDist. Our harness traces a Pareto hull but does not implement the benchmark's exact definition of that metric, and an omitted number beats an invented one.

---

API

Input

Output

out["ranking"]      # top-5, best first
                    #   [{"slug": "gpt-5-6-sol",             "score": 1.000},
                    #    {"slug": "gemini-3-5-flash-medium", "score": 0.332},
                    #    {"slug": "gemini-3-1-pro-preview",  "score": 0.285}, ...]
out["confidence"]   # 0.966  — how sure the classifier was about the request type
out["why"]          # "Selected gpt-5-6-sol for a performance cost/capability trade-off."

out["ranking"][0]["slug"] is the model to call. score is each candidate's share of the winner's probability, so it is bounded (0, 1] and tells you how clear-cut the pick was — a runner-up at 0.9 means either would do, at 0.05 it would not.

All three presets at once, with the ordering guaranteed

out = r.route_ladder(messages)
out["economy"]["ranking"][0]["slug"]
out["performance"]["ladder_repaired"]   # True if monotonicity repair moved this pick

Prefer this whenever you show the three presets as an ordered choice. Calling route() once per preference gives an ordering that holds only on average — we measured 3.27% of prompts inverting on price and 2.07% on capability, because the ranking objective this model was trained against is not itself monotone in the preference. If you render three tiers to a user, a few percent of them will show "Economy" costing more than "Balanced".

route_ladder() enforces the ordering per request (0 violations, asserted at runtime), fires on ~2–4% of prompts, and runs the encoder once instead of three times. Balanced is the anchor and is byte-identical to route(preference="balanced").

Batching

outs = r.route_batch([messages_a, messages_b], preference="economy")

---

What's in the box

Architecture: d_query=320, d_ship=32, 30 candidates, 7 classes, max_seq_length=8192, last-token pooling with right padding, opset 17.

---

Deliberate design decisions

We'd rather document these than have you discover them.

• You get a ranking, not a scorecard. The response is slugs and relative scores. The candidate vectors in the bundle are a learned projection you cannot read them back out of. If you need cost attribution, join the chosen slug against your own pricing table.
• The roster is fixed at 30 models. Candidate vectors come from our feature pipeline, so you cannot add or swap models yourself — tell us and we ship an updated bundle. If you need a larger or bespoke roster, that is what Pro is for.
• Capability classes are unlabelled. The 7 classes surface as label-0 … label-6. They are stable across releases, so you can key logic on them, but we do not publish what each one means.
• int8 quantisation is deliberately not shipped. It cut size 4× but changed the chosen model on up to 18% of prompts for ~14% less latency. A ranking that is "close" is a different answer.
• CPU latency is ~190 ms p50 per prompt on a shared 12-core VM, dominated by the encoder. Batch with route_batch where you can.
• Pooling is last-token with right padding. If you write your own runtime, pool at attention_mask.sum(1) - 1, not [-1] — the latter silently pools a pad token for every sequence shorter than the batch maximum, and batch size 1 hides it.

---

Attribution

The encoder is a fine-tuned derivative of codefuse-ai/F2LLM-v2-80M (Apache-2.0), further trained by distillation from an in-house teacher with a 7-class head added. All weights differ from upstream. See NOTICE.

_{primitive · Apache-2.0}