ZOrch-1 Triage (Julia-1 fine-tune)
Routes a short, noisy or multilingual ticket to three labels in one pass:
| Head | Labels |
|---|---|
| role (12) | product, business_analysis, research, design, frontend, mobile, backend, data, ai, qa, devops, security |
| action (11) | implement, document, fix, test, research, review, analyze, investigate, design, deploy, refactor |
| risk (5) | normal, production, security, data_sensitive, architecture |
Base model: SupersonicLabs/Julia-1 (about 144M parameters), full fine-tune, bf16, one RTX 4090. Inputs may contain only a title, or a title with description, labels and Given/When/Then criteria, in English, Vietnamese (with or without diacritics) or Korean.
Honest results
Held-out real benchmark, 329 tasks never used for training or data generation. "Confident labels" is the 134-task subset where the benchmark's own annotators agreed.
| Subset | role | action | risk | all three correct |
|---|---|---|---|---|
| All tasks (329) | 64.4% | 82.1% | 84.8% | 47.1% |
| Confident labels (134) | 79.9% | 82.1% | 90.3% | 62.7% |
The earlier ZOrch v3 report on the same kind of data scored 42% on all three heads, so this is a modest improvement, not a solution. The 95% target was not reached. Four training rounds (more synthetic data, targeted contrast data, higher real-data weight, higher learning rate) all landed between 44% and 47% on all tasks. A blind LLM labeller following the same label guide agreed with the benchmark on 93% of roles and 81.8% of full triples (98.5% on confident labels), so the labels are learnable, but this 144M encoder does not generalise to them yet.
Confidence gating
Use needs_review (lowest head confidence below a threshold) to route uncertain tickets to a person.
| Min confidence | Coverage, confident labels | All 3 correct | Coverage, all tasks | All 3 correct |
|---|---|---|---|---|
| 0.80 | 46% | 88.5% | 36% | 73.1% |
| 0.90 | 25% | 90.9% | 18% | 80.0% |
| 0.95 | 14% | 100% (n=19) | 10% | 87.5% (n=32) |
Small samples: treat the 0.95 figures as indicative only.
Weak spots
- Role is the bottleneck:
dataandsecurityare confused withbackend,devopsandai. - Vietnamese tasks: action 73.3% (n=30).
- Short, context-free titles score lower than Gherkin-style tickets (55.9% vs 69.7% all three, confident labels).
Training data
About 6,200 synthetic tickets written by LLM agents from a label guide plus real few-shot examples, kept only where an independent blind labeller agreed per head, then augmented (diacritic stripping, typos, truncation, dropped fields). Real labelled tickets were used for training only from a separate split. No real benchmark rows are published.
Usage
git clone https://github.com/niko0xdev/julia-1-zorch && cd julia-1-zorch && pip install -e .
python scripts/v3/predict.py <model_dir> '{"title": "dbt model cho bao cao doanh thu"}' --threshold 0.9
Source, builders and the evaluation script: https://github.com/niko0xdev/julia-1-zorch (scripts/v3/).