Sieve-2B-v3

Sieve-2B-v3 is a decision model from the Sieve family. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a full fine-tune of Qwen/Qwen3.5-2B (revision 15852e8c) with a pointer head, and it stores one calibration temperature per question category. The code is at github.com/sthanika-ai/Sieve.

Summary

How it works

• Backbone. Only the text part of Qwen3.5-2B is used; the model cannot produce text.
• Full fine-tune. All 24 decoder layers were trained; the token embeddings and the final norm are the backbone's. The release stores the trained layers in full as a PEFT adapter (modules_to_save), with a zero rank-1 LoRA on the embeddings, so it loads like the LoRA models of the family.
• Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the probabilities.
• Isolation. Each question runs as its own row from the state's cache, so questions never see each other.

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused kernel our runs used

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-2B-v3")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

• Loading. Load it with sieve 0.3.0 or later; the adapter alone is not a text-generation model. The backbone, Qwen/Qwen3.5-2B, is downloaded on first use; our runs used revision 15852e8c.
• GPU memory. In bf16, 4.3 GB was in use on the CUDA-graph server and 10.7 GB on the eager server during our latency runs. Longer states need more.
• Token limits. States and questions up to 16,384 tokens each, as in training; sieve reads these limits from training_config.json. Longer requests are refused, never truncated. The Decision Index engine declares larger limits (a 65,536-token state and a 32,768-token question), and every request of the Decision Index run below fit within them.
• Server. sieve-serve --model sthanika-ai/Sieve-2B-v3 --graphs.
• Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-2B-v3 --out runs/Sieve-2B-v3.
• causal-conv1d. Without it, transformers falls back to a slower PyTorch convolution, whose probabilities can differ slightly. Install it to reproduce our numbers.
• Temperature per category. head.pt stores a global T = 0.968 and 25 temperatures (0.869 to 1.257), fitted by NLL on the calibration split (143,822 questions). A question's category is its type, its number of options (2, 3-4, 5-9, 10-25, 26+) and the kind of state (empty, text or JSON). A question takes the temperature of its most specific category in the table (choice:3-4:text, then choice:3-4, then choice), else the global T. Temperatures never change an answer, and temperature=1.0 gives raw probabilities.

Results

Decision Index

Sieve-2B-v3 scores 53.51 (raw index 65.39, breadth 51.09) on the public 0.3 suite of the Decision Index, on our own run with the official kit at f5a6e4f (0.3) and its scorer.

• Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
• How the run was made. One complete run of the 0.3 suite on two A100 80GB PCIe GPUs, in six shards.
• Results. sthanika-ai/Sieve-2B-v3-decision-index-results.
• Not on the leaderboard yet. The score is self-reported. The board ranks by the Full score: the public benchmarks count 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

All 43 benchmarks

Calibration

The calibration split chose the checkpoint and fitted the temperatures.

In a 2-fold check inside the calibration split (fit on one half, score the other), the per-category table scored NLL 0.3011 and ECE 0.0036, against 0.3014 and 0.0046 for a single T.

Latency

• Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of the 0.3 run (A100 80GB PCIe): median 59 ms, mean 143 ms, 80th percentile 115 ms, 95th percentile 329 ms. Six engines (three per GPU) ran at once, next to other users' GPU jobs, so these are upper bounds. This is not the maintainers' RTX PRO 6000 measurement.
• Cached state. Server-reported p50 on one A100 80GB PCIe (bf16, merged, CUDA graphs), for one choice question on a 1,433-token state:

• By option count. For one choice question on a 41-token state, which is below the server's 64-token caching threshold, so every request ran the state again. Server p50 in ms, CUDA graphs / eager:

Training

• Recipe.
  • Full fine-tune of the 24 decoder layers (1,373,263,680 parameters); the token embeddings and the final norm stay frozen. The pointer head (256-dimensional projections, fp32) is trained from scratch, alone for the first 100 steps.
  • AdamW, lr 1e-5 for the backbone and 5e-5 for the head, weight decay 0.01 on weight matrices, gradient clipping at 1.0, 5% linear warm-up then cosine decay to 0.1×, 1 epoch, 128 records per step.
  • bf16 compute with fp32 master weights, gradient reduction and optimizer state; FSDP2 with one unit per decoder layer; gradient checkpointing.
  • States and questions up to 16,384 tokens each; longer records are dropped, never truncated (none were).
  • 3,344 steps, 10.6 hours on two A100 80GB PCIe GPUs.
  • Permuted choice options, "none of the above" augmentation (as the answer, as a wrong option and in minimal pairs) and irrelevant distractor options, with a cross-entropy loss.
• Selection. The checkpoint with the highest accuracy on a fixed 2,004-question calibration sample, with NLL breaking ties. This rule was fixed before training, and it chose step 3,300. The benchmarks were never used to choose.
• Temperatures. Fitted after selection on the whole calibration split: the global T, then one T per category with at least 200 calibration questions.
• Data. 428,018 training records, with 71,018 for calibration and 215,392 for test (60/10/30).
  • Sources. Public datasets and generated decision records.
  • Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Limitations

• score questions. The training data has only choice and noul questions. score questions are answered by the same pointer head, but the model was not trained on them, and they take the global temperature.

Files

License

Apache-2.0, see LICENSE.