julia-1-router

julia-1-router is a prompt router. Given a user prompt, it picks one model tier: small, medium, large or reasoning. It is a full fine-tune of niko0xdev/julia-1-zorch-triage, built on SupersonicLabs/Julia-1. It has a single decision head, route.

Status: target met on synthetic held-outs. On the fixed synthetic held-out sets, accuracy is above 95% and under-routing is below 2% on both heldout_combo and heldout_style. All evaluation here is synthetic; there are no real user prompts in the metrics below.

Tiers

Tie-break policy used for labeling: when a task sits between two tiers, it gets the higher one.

Training

• Format: trainer rows {state, question, options, target}. The question is Which model tier should handle this prompt? and the options are ["small", "medium", "large", "reasoning"]. The prompt goes in state.title, with any attachment or context in state.description.
• Data: rt008 train split, 60,000 prompts: 15,000 per tier; vi 18,000, vi_nodiac 15,000, ko 15,000, en 12,000. They come from 174 task families. Held-out sets are unchanged copies of rt001 held-outs.
• Init: continued from the rt007 best checkpoint (itself continued from rt005 best over the earlier public rt002 weights).
• Selection: checkpoint chosen on the in-distribution dev split (1,200 prompts) by task_exact_match. The selected checkpoint is at optimizer step 500 (dev task exact match 1.0). That score is in-distribution and not a held-out result.
• Hyperparameters (training_config.yaml): 1 epoch, learning rate 1e-5, micro batch 1, gradient accumulation 16, warmup 0.08, weight decay 0.01, max length 256, evaluation every 500 steps, seed 20261208.
• Hardware: Google Colab NVIDIA L4.
• Inference policy: raw argmax over the four route logits. A confidence-based escalate-one-tier policy was tried on train-side OOD/dev only (never on held-out) and was not adopted, because the remaining misses were often confidently wrong or skipped more than one tier. See the dataset card and CONFIDENCE_ESCALATE.md in the training repo for the sweep notes.

Iteration history

Held-out sets were never edited. Train data was widened from held-out error analysis only (new train-only families; no held-out prompts or templates copied into train). Leak checks (family / title / style / fulltext overlap) were run each round.

Evaluation (synthetic held-out sets)

There are no real user prompts in any evaluation. Both held-out sets are synthetic and frozen since rt001:

• heldout_combo: 20 task families (5 per tier) and the held-out 20% of every slot pool. None of these appear in training. Train-time wrapper styles are used.
• heldout_style: the same held-out families and pool items, wrapped in 5 styles that never appear in training.

Cost weights small 1, medium 3, large 10, reasoning 15 are assumed relative costs, not measured prices.

• Cost ratio vs gold: the total assumed cost of the predicted tiers divided by the total cost of the gold tiers.
• Overspend: the extra cost from over-routing, as a share of the gold cost.
• Under-routing: the predicted tier is cheaper than the gold tier.
• Over-routing: the predicted tier is more expensive than the gold tier.

The target (accuracy above 95% and under-routing below 2% on each held-out set) was met on rt008.

Per-tier accuracy, heldout_combo: small 97.7% / medium 96.7% / large 99.7% / reasoning 99.7%. heldout_style: small 87.0% / medium 97.7% / large 99.3% / reasoning 100.0%.

Per-language accuracy, heldout_combo: en 99.2% / ko 99.7% / vi 98.3% / vi_nodiac 96.7%. heldout_style: en 97.9% / ko 95.0% / vi 96.1% / vi_nodiac 95.3%.

Weakest held-out families, heldout_combo: study_plan 92%, dockerfile 93%, yes_no_fact 93%. heldout_style: yes_no_fact 78%, word_count 83%, synonym 86%, timezone 93%.

Confusion matrix, heldout_combo (rows gold, columns predicted: small, medium, large, reasoning):

Confusion matrix, heldout_style:

Recommended use

• Default inference: argmax over the four route probabilities. No confidence escalate is applied in the published checkpoint.
• For high-stakes prompts where over-routing is cheaper than under-routing, you may still escalate one tier when the top probability is low. That policy was not tuned on held-out data and is optional.
• Remaining misses on heldout_style are mostly small-tier prompts (yes_no_fact, word_count, synonym) over-routed to medium or reasoning.

Usage

Load with the julia-1-zorch / Julia-1 routing stack (same layout as other julia-1 classification fine-tunes: model.safetensors, encoder/, tokenizer/, julia_config.json). Example sketch:

# Pseudocode: use your project's router load + predict helpers.
# pred = route_prompt(model, tokenizer, title=prompt, description=optional_context)
# pred in {"small", "medium", "large", "reasoning"}

Limitations

• Metrics are on synthetic held-outs only. Real traffic may differ.
• heldout_style small-tier accuracy is 87.0%; style shift still hurts some short factual prompts.
• Assumed tier costs are placeholders, not vendor prices.
• The public GitHub tree may lag the local router training scripts used for this run.

Files

• model.safetensors: fine-tuned weights (selected rt008 best)
• encoder/, tokenizer/, julia_config.json: runtime config matching Julia-1 classification layout
• training_config.yaml, training_history.json, validation_metrics.json: training record
• heldout_eval.json: full held-out metrics for rt008

Citation

If you use this model, please cite the Hugging Face repo niko0xdev/julia-1-router and the dataset niko0xdev/julia-1-router-data.