julia-1-zorch
A compact SDLC control-plane classifier that reads a JIRA ticket and decides who owns it, what happens next, which review gate applies, who executes, how risky it is, and whether a human must approve.
v2: robust to missing fields, casing, truncation and typos.
Result. 100.00% decision accuracy on the held-out test split (base
SupersonicLabs/Julia-1: 25.06%) and 100.00% on 5 unseen, de-duplicated suites (26,400 decisions). Under 13 input perturbations the mean accuracy is 99.0% (v1: 84.8%), and the worst case, only one content field kept, is 87.1% (v1: 57.6%).
What changed in v2
v1 scored 99.7 % on clean tickets but collapsed when the Gherkin criteria were removed (about 79 %), when text was upper-cased (about 70 %) or truncated (about 81 %). v2 is trained on the same six decisions with a larger, de-duplicated dataset and label-preserving input augmentation, so the model no longer depends on any single field or on surface formatting.
| v1 | v2 | |
|---|---|---|
| Training records | 3,720 clean | 28,560 (14,280 clean + 14,280 perturbed) |
| Augmentation | none | field dropout, casing, truncation, typos, whitespace |
| Clean held-out accuracy | 99.74% | 100.00% |
| Mean accuracy under perturbation | 84.8% | 99.0% |
| Worst-case perturbation | 57.6% | 87.1% |
What it decides
| Decision | Question | Label space |
|---|---|---|
role_owner |
Which SDLC role should own the next unit of work? | product_manager, business_analyst, solution_architect, ux_designer, frontend_engineer, backend_engineer, mobile_engineer, qa_engineer, devops_engineer, security_engineer |
next_action |
What should happen next in the SDLC? | clarify_requirements, architecture_review, design_ui, implement, write_or_update_tests, security_review, deploy_or_configure, monitor_or_observe, investigate_incident, release_review |
review_gate |
Which review gate is required before this work can progress? | none, peer_review, architecture_review, qa_review, security_review, product_acceptance, change_advisory |
executor_profile |
Which executor profile should handle the next action? | product_agent, analysis_agent, architecture_agent, coding_agent, qa_agent, devops_agent, security_agent, human_only |
risk_level |
What is the delivery risk level of this change? | low, medium, high, critical |
human_required |
Is explicit human approval required before autonomous progression? | no, yes |
Results
Held-out test split
774 decisions from 129 scenarios not used for training or model selection.
| Metric | Fine-tuned | Base | Change |
|---|---|---|---|
| Accuracy | 100.00% | 25.06% | +74.94 pts |
| Negative log-likelihood | 0.0000 | 3.6520 | -3.6520 |
| Expected calibration error | 0.0000 | 0.4740 | -0.4740 |
| Decision | Fine-tuned | Base |
|---|---|---|
| Role owner | 100.00% | 12.40% |
| Next action | 100.00% | 17.83% |
| Review gate | 100.00% | 17.05% |
| Executor profile | 100.00% | 15.50% |
| Risk level | 100.00% | 35.66% |
| Human approval | 100.00% | 51.94% |
Unseen multi-seed suites
5 suites generated with seeds 101, 202, 303, 404, 505, none used in training. Each record was hashed and checked against the training, validation and test data first (0 exact duplicates found).
| Suite | Decisions | Accuracy | NLL | ECE |
|---|---|---|---|---|
seed_101 |
5,280 | 100.00% | 0.0000 | 0.0000 |
seed_202 |
5,280 | 100.00% | 0.0000 | 0.0000 |
seed_303 |
5,280 | 100.00% | 0.0000 | 0.0000 |
seed_404 |
5,280 | 100.00% | 0.0000 | 0.0000 |
seed_505 |
5,280 | 100.00% | 0.0000 | 0.0000 |
| Pooled | 26,400 | 100.00% |
| Decision | Pooled accuracy |
|---|---|
| Role owner | 100.00% |
| Next action | 100.00% |
| Review gate | 100.00% |
| Executor profile | 100.00% |
| Risk level | 100.00% |
| Human approval | 100.00% |
Robustness to degraded input
One unseen suite (seed_505, 1,320 decisions) is re-scored after perturbing every ticket. Trained families are the augmentations used in training; held-out families were never seen, so they show how well the robustness transfers.
| Perturbation | Family | v1 | v2 | Change |
|---|---|---|---|---|
| clean (reference) | 99.92% | 100.00% | +0.08 pts | |
| UPPERCASED | trained | 54.39% | 100.00% | +45.61 pts |
| only one content field kept | trained | 57.58% | 87.12% | +29.55 pts |
| no Gherkin / acceptance criteria | trained | 69.24% | 100.00% | +30.76 pts |
| Title Case | trained | 80.83% | 100.00% | +19.17 pts |
| text truncated (30-150 chars) | trained | 84.85% | 100.00% | +15.15 pts |
| one content field removed | trained | 86.14% | 100.00% | +13.86 pts |
| 3% character typos | trained | 90.15% | 99.85% | +9.70 pts |
| lowercased | trained | 98.11% | 100.00% | +1.89 pts |
| 1-3 metadata fields removed | trained | 99.47% | 100.00% | +0.53 pts |
| whitespace collapsed | trained | 99.62% | 100.00% | +0.38 pts |
| word dropout (20%) | held-out | 83.71% | 99.70% | +15.98 pts |
| shuffled lines | held-out | 98.26% | 100.00% | +1.74 pts |
| [URGENT] prefix | held-out | 99.47% | 100.00% | +0.53 pts |
Confidence and selective prediction
| Threshold | Coverage | Accuracy on covered |
|---|---|---|
| 0.50 | 100.00% | 100.00% |
| 0.70 | 100.00% | 100.00% |
| 0.80 | 100.00% | 100.00% |
| 0.90 | 100.00% | 100.00% |
| 0.95 | 100.00% | 100.00% |
Suggested policy: auto-accept at confidence >= 0.90 and escalate the rest to a stronger judge or a human.
Limitations
- Synthetic data. Training and every evaluation set come from one scenario generator. The numbers are in-distribution; they are not a claim about real JIRA projects. Validate on reviewed tickets from your own backlog before relying on the model.
- Weakest case. only one content field kept is the hardest perturbation at 87.1%; tickets that lose that much information should be escalated.
- Two ticket formats. Training covers Gherkin-style tickets (
background,feature,gherkin) and classic JIRA tickets (summary,description,acceptance_criteria). Other layouts are untested. - Scope. English only, six fixed decisions, 1,024-token window. It is a classifier: it does not plan or explain.
Inference
The weights use a custom JuliaDecisionModel head on top of the base model. The inference code is part of the companion zorch project that produced this release; this repository holds the weights, configuration and evaluation artefacts. Input is a ticket state object plus the decision to ask; the output is a probability distribution over that decision's labels.
{
"state": {
"key": "CORE-1000",
"product": "Mobile",
"component": "web-shell",
"feature": "Define success criteria for Mobile usage limits",
"labels": ["web-shell", "github-actions", "medium"],
"priority": "Medium",
"technology_context": "GitHub Actions",
"workflow_status": "Backlog",
"background": "Component web-shell uses GitHub Actions; priority is Medium.",
"gherkin": "Given multiple customer plans have different limits\nWhen the team prepares the next release\nThen the exact limit policy and success metric must be agreed before implementation"
}
}
Training
| Method | Full fine-tune, all weights unfrozen |
| Trainable parameters | 144,192,769 of 144,292,867 (99.93%) |
| Data | 14,280 clean records (original train + two extra seeds) plus one perturbed copy of each; no overlap with any evaluation set |
| Augmentation | drop Gherkin / acceptance criteria, drop a content field, keep a single field, drop 1-3 metadata fields, lower / UPPER / Title case, truncate to 30-150 characters, 3 % character typos, whitespace collapse (one or two per copy, labels unchanged) |
| Validation | original validation split, clean plus one perturbed copy; checkpoint chosen on NLL |
| Epochs | 2 |
| Effective batch size | 16 (8 x 2) |
| Learning rate | 2e-05, warm-up 0.05, weight decay 0.01 |
| Precision | bf16 |
| Wall-clock | 9.8 min on 1 x NVIDIA RTX 4090 (24 GB) |
| Best validation NLL | 0.0113 |
| Seed | 17 |
Files
| File | Purpose |
|---|---|
model.safetensors, julia_config.json, encoder/, tokenizer/ |
Weights and configuration |
eval/suite_metrics.json, eval/stress_compare.json |
Multi-seed and robustness results (v2 vs v1) |
test_metrics.json, test_predictions.json |
Held-out metrics and per-record predictions |
training_config.yaml, resolved_config.json, history.json, training_summary.json |
Training provenance |
dataset_generation_report.json, dataset_lint_report.json, TAXONOMY.md |
Data provenance |
Built on SupersonicLabs/Julia-1. Apache-2.0; the base model's terms also apply.