Vibe Router v3 — ModernBERT
A tiny LLM router that decides whether a chat request should run locally (on-device) or in the cloud, built on ModernBERT-base.
How it works
Given a user prompt, the model outputs a single logit. After sigmoid, values above the threshold route to cloud; below routes to device.
Recommended thresholds
The optimal threshold depends on your use case. Higher thresholds send more traffic to the device model, saving cost and latency at the expense of quality.
Start with threshold=0.95 for a good balance between quality and cost savings. Adjust based on your device model's capabilities.
Training
Fine-tuned end-to-end from answerdotai/ModernBERT-base using Privileged Information Distillation (PID) loss on 49,700 labeled prompt pairs with soft teacher labels derived from dual-judge pairwise comparison (GPT-4o + Claude Sonnet 4).
HP sweep results
Performance
Baselines
Threshold sweep (test set)
Latency
Usage
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch
model_id = "darkolorin/vibe-router-modernbert-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=1)
model.eval()
prompt = "Write a Python B-tree implementation"
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
logits = model(**inputs).logits
p_cloud = torch.sigmoid(logits).item()
threshold = 0.95 # recommended balanced threshold
decision = "cloud" if p_cloud > threshold else "device"
print(f"p(cloud)={p_cloud:.3f} → {decision}")
Routing examples (threshold=0.95)
Dataset
- 49,700 samples from diverse HuggingFace conversation datasets
- Both models (LFM2.5-1.2B-Instruct and GPT-5.2) generate responses for each prompt
- Dual-judge pairwise comparison: GPT-4o and Claude Sonnet 4 compare outputs side-by-side
- Soft teacher labels via win-rate aggregation with temperature τ=0.2
- 96.1% cloud-preferred, reflecting genuine capability gap between 1.2B and GPT-5.2
Changes from v1
License
Apache 2.0
Citation
@misc{vibe-router-2026,
title={Vibe Router: On-Device LLM Routing with Privileged Information Distillation},
author={Mirai},
year={2026},
url={https://github.com/trymirai/vibe_router}
}