Sieve-27B-v2

Sieve-27B-v2 is a decision model. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a full fine-tune of Qwen/Qwen3.8-27B (revision 1d4bf0f2) with a pointer head, and it stores one calibration temperature per question category. The code is at github.com/sthanika-ai/Sieve.

Summary

How it works

• Backbone. Only the text part of Qwen3.8-27B is used; the model cannot produce text.
• Full fine-tune. All 64 decoder layers were trained; the token embeddings and the final norm are the backbone's. The release stores the trained layers in full as a PEFT adapter (modules_to_save), with a zero rank-1 LoRA on the embeddings. sieve 0.3.2 and later copy these layers into the backbone in place, so the model needs about the backbone's memory to load.
• Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the probabilities.
• Isolation. Each question runs as its own row from the state's cache, so questions never see each other.

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # optional on A100 / H100: the fused convolution kernel

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-27B-v2")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

• Loading. Load it with sieve 0.3.2 or later, which loads a full fine-tune in place; the adapter alone is not a text-generation model. The backbone, Qwen/Qwen3.8-27B, is downloaded on first use; our runs used revision 1d4bf0f2.
• GPU memory. In bf16, 50.3 to 53.1 GB was in use on the CUDA-graph server and 52.2 GB on the eager server during our latency runs. One 80 GB GPU is enough; longer states need more.
• Token limits. States and questions up to 16,384 tokens each, as in training; sieve reads these limits from training_config.json. Longer requests are refused, never truncated. The Decision Index engine declares larger limits (a 65,536-token state and a 32,768-token question), and every request of the Decision Index run below fit within them.
• Server. sieve-serve --model sthanika-ai/Sieve-27B-v2 --graphs.
• Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-27B-v2 --out runs/Sieve-27B-v2.
• causal-conv1d. Our training, Decision Index and latency runs were on NVIDIA B300 GPUs, for which causal-conv1d has no kernels, so they used transformers' PyTorch convolution. On A100 or H100 the fused kernel is faster; its probabilities can differ slightly from ours.
• Temperature per category. head.pt stores a global T = 0.999 and 25 temperatures (0.928 to 1.127), fitted by NLL on the calibration split (132,395 questions, outside the checkpoint-selection sample). A question's category is its type, its number of options (2, 3-4, 5-9, 10-25, 26+) and the kind of state (empty, text or JSON). A question takes the temperature of its most specific category in the table (choice:3-4:text, then choice:3-4, then choice), else the global T. Temperatures never change an answer, and temperature=1.0 gives raw probabilities.

Results

Decision Index

Sieve-27B-v2 scores 68.02 (raw index 75.61, breadth 67.38) on the public 0.3 suite of the Decision Index, on our own run with the official kit at f5a6e4f (0.3) and its scorer.

• Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
• How the run was made. One complete run of the 0.3 suite on two NVIDIA B300 GPUs, in six shards.
• Results. sthanika-ai/Sieve-27B-v2-decision-index-results.
• Not on the leaderboard yet. The score is self-reported. The board ranks by the Full score: the public benchmarks count 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

All 43 benchmarks

Calibration

The calibration split chose the checkpoint and fitted the temperatures.

In a 2-fold check inside the calibration split (fit on one half, score the other), the per-category table scored NLL 0.1875 and ECE 0.0026, against 0.1876 and 0.0029 for a single T.

Latency

• Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of the 0.3 run (NVIDIA B300): median 136 ms, mean 347 ms, 80th percentile 322 ms, 95th percentile 918 ms. Six engines (three per GPU) ran at once, so these are upper bounds. This is not the maintainers' RTX PRO 6000 measurement.
• Cached state. Server-reported p50 on one NVIDIA B300 (bf16, CUDA graphs), for one choice question on a 1,433-token state:

• By option count. For one choice question on a 41-token state, which is below the server's 64-token caching threshold, so every request ran the state again. Server p50 in ms, CUDA graphs / eager:

Training

• Recipe.
  • Full fine-tune of the 64 decoder layers (24,353,196,544 parameters); the token embeddings and the final norm stay frozen. The pointer head (256-dimensional projections, fp32) is trained from scratch, alone for the first 100 steps.
  • AdamW, lr 4e-6 for the backbone and 5e-5 for the head, weight decay 0.01 on weight matrices, gradient clipping at 1.0, 5% linear warm-up then cosine decay to 0.1×, 1 epoch, 128 records per step.
  • bf16 compute with fp32 master weights, gradient reduction and optimizer state; FSDP2 with one unit per decoder layer; gradient checkpointing.
  • States and questions up to 16,384 tokens each; longer records are dropped, never truncated (none were).
  • 3,133 steps, 27.55 hours (55.1 GPU-hours) on two NVIDIA B300 GPUs.
  • Permuted choice options, "none of the above" augmentation (as the answer, as a wrong option and in minimal pairs) and irrelevant distractor options, with a cross-entropy loss.
• Selection. The checkpoint with the highest accuracy on a fixed 6,004-question calibration sample, with NLL breaking ties. This rule was fixed before training, and it chose the last step, 3,133. The benchmarks were never used to choose.
• Temperatures. Fitted after selection on the calibration split outside the selection sample: the global T, then one T per category with at least 200 calibration questions.
• Data. 400,985 training records, with 65,959 for calibration and 201,027 for test (60/10/30).
  • Sources. Public datasets and generated decision records.
  • Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Limitations

• score questions. The training data has only choice and noul questions. score questions are answered by the same pointer head, but the model was not trained on them, and they take the global temperature.

Files

License

Apache-2.0, see LICENSE.