Darwin-27B-ZTC-v9

Darwin-27B-ZTC-v9 is a decision model. It does not generate text. Given a state and a typed question, it returns a calibrated probability for every option in one forward pass: yes or no (noul), one of several options (choice), or a level on an ordered scale (score). Requests use the System One shape (state plus questions), so clients written for that format work unchanged.

It is the next version of Darwin-27B-ZTC-v5. The main change is much stronger multiple-choice decisions (intent, entity type, relation, part of speech, rule reasoning, product relevance) while keeping the yes/no and score behaviour of v5.

Results

S1MB (System One Mosaic Benchmark), english-v1

All 137 benchmarks complete, measured with the S1MB evaluator and its unchanged autojev adapter, zero failed decisions. Merged by the maintainer on 2026-10-19 (PR #75). The ranking below follows the leaderboard's own Borda method (viewer/src/lib/borda.ts), recomputed on the published results of 2026-10-19 (195 complete models) with this model added; the live leaderboard is authoritative. At submission (results of 2026-10-17, 171 models) it also ranked first, with 91.29.

Largest gains over v5 (baseline-adjusted score)

Training and data

Full-weight fine-tuning in the Darwin-27B-ZTC line on public train splits of decision datasets under licenses that allow commercial use. For v9 we added the original upstream train splits of choice-task sources (for example DBpedia-14, SNLI, MASSIVE, HWU64, BANKING77, CLINC150, MTOP, Few-NERD, UD English-EWT, Gretel PII masking, ESCI, WIQA, FOL-NLI, BoardgameQA, IBM argument quality), converted to the System One format.

• No test split of S1MB or of any benchmark was used for training.
• Every training row was screened against all S1MB cases (exact match of the normalised state, rare long text fields and duplicate states); matches were removed. Converters were checked by reproducing the S1MB test cases from the upstream test splits, and those test splits were used only for that check.
• Inference is one forward pass per question with the checkpoint temperature; no ensembling.

Usage

Python 3.12, a CUDA GPU with about 60 GB of free memory (BF16 weights), and the code shipped in this repository (autojev/).

import sys
from pathlib import Path
from huggingface_hub import snapshot_download

path = Path(snapshot_download("FINAL-Bench/Darwin-27B-ZTC-v9"))
sys.path.insert(0, str(path))
from autojev.model import DecisionModel, answer

model = DecisionModel(checkpoint=path).eval()
question = {
    "type": "choice",
    "instructions": "Identify the banking intent expressed by the utterance.",
    "criteria": {"card_arrival": "Card delivery timing", "lost_or_stolen_card": "Card lost or stolen", "top_up_failed": "Top-up failed"},
}
row = {"state": {"utterance": "How long does a new card take to arrive?"}, "question": question}
print(answer(question, model.predict([row])[0]))

ztc_server.py serves POST /v1/systemone with batching.

curl http://localhost:8000/v1/systemone -H "Content-Type: application/json" -d '{"model": "FINAL-Bench/Darwin-27B-ZTC-v9",
  "state": "I was charged twice for my order and want a refund.",
  "questions": {
    "refund": {"type": "noul", "instructions": "Is a refund requested?"},
    "intent": {"type": "choice", "instructions": "What does the customer want?",
               "criteria": {"refund": "wants money back", "cancel": "wants to cancel an order", "track": "wants to track an order"}}}}'

License

Apache-2.0. The decision code in autojev/ is MIT (upstream autojev by Denis Yarats) with changes for text-only backbones. Training data keep their own licenses; several sources are CC BY-SA (DBpedia-14, SNLI, UD English-EWT, MTOP, Few-NERD) and are credited here.