Sieve-27B
Sieve-27B is a decision model from the Sieve family. You give it a piece of text
or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question.
It reads the state once, scores the options directly, and never generates text.
It is a LoRA adapter (rank 32) and a pointer head on Qwen/Qwen3.8-27B
(revision 1d4bf0f2). The code is at github.com/sthanika-ai/Sieve.
Summary
How it works
- Backbone. Only the text part of Qwen3.8-27B is used; the model cannot produce text.
- Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the
probabilities.
- Isolation. Each question runs as its own row from the state's cache, so questions never see each other.
Use
pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0 # recommended: the fused kernel our runs used
from sieve import load_sieve, decide
m = load_sieve("sthanika-ai/Sieve-27B")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
{"department": {"type": "choice", "instructions": "Which team handles this?",
"criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
"churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})
- Loading. Load it with
sieve; the adapter alone is not a text-generation model. The backbone is downloaded at
its pinned revision on first use.
- GPU memory. About 55 GB in bf16 (52,683 MiB in use after loading), so one 80 GB GPU is enough.
- Token limits. States and questions up to 16,384 tokens each, as in training.
sieve 0.2.1 or later uses these
limits by default; earlier versions stop at 8,192 tokens. Longer requests are refused, never truncated. The Decision
Index engine declares larger limits (a 65,536-token state and a 32,768-token question). In the Decision Index run
below, 241 requests had a question longer than 16,384 tokens (at most 29,944, from ToolRet and BRIGHT) and were
answered in full; no state was longer than 16,384.
- Server.
sieve-serve --model sthanika-ai/Sieve-27B --graphs.
- Decision Index.
python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-27B --out runs/Sieve-27B.
- causal-conv1d. Without it, transformers falls back to a slower PyTorch convolution, whose probabilities can
differ slightly. Install it to reproduce our numbers.
- Temperature.
head.pt stores T = 1.050, fitted by NLL on the calibration split (43,309 questions). T never
changes an answer, and temperature=1.0 gives raw probabilities.
Results
Decision Index
On the Decision Index leaderboard (edition 0.3.2,
board data of 2026-10-12), Sieve-27B has a Full score of 60.17 and a public score of 60.04. Our own run of the
public 0.3 suite, with the official kit at 62d2f51 (0.3) and its
scorer, scores 60.04 (raw index 69.32, breadth 58.67).
- Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported
request and no error. The index averages 37 of the benchmarks.
- How the run was made. One complete run of the 0.3 suite on two A100 80GB PCIe GPUs, in two shards.
- Results. sthanika-ai/Sieve-27B-decision-index-results.
- Full score. The board ranks by the Full score: the public benchmarks count 20%, the maintainers' private tests
of the same skills 50% and their private tasks from new domains 30%.
The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each
benchmark's own metric and the skill is chance-corrected.
All 43 benchmarks
Calibration
The calibration split chose the checkpoint and fitted the temperature.
Latency
- Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of
the 0.3 run (A100 80GB PCIe): median 151 ms, mean 416 ms, 80th percentile 425 ms, 95th percentile 966 ms. This is
not the maintainers' RTX PRO 6000 measurement.
- Cached state. Server-reported p50 on one A100 80GB PCIe (bf16, merged, CUDA graphs), for one choice question on a
1,433-token state:
- By option count. For one choice question on a 41-token state, which is below the server's 64-token caching
threshold, so every request ran the state again. Server p50 in ms, CUDA graphs / eager:
Training
- Recipe.
- LoRA r=32, α=64, dropout 0.05 on the attention, MLP and linear-attention projections (12 modules), with the
pointer head trained from scratch.
- AdamW, lr 5e-5 for the adapter and the head, OneCycle with 10% warm-up, 1 epoch, 32 records per step, bf16.
- States and questions up to 16,384 tokens each.
- 4,813 steps, about 27 hours on two A100 80GB PCIe GPUs.
- Permuted choice options and "none of the above" augmentation, with a cross-entropy loss.
- Selection. The checkpoint with the highest accuracy on a fixed 2,001-question calibration sample, with NLL
breaking ties. This rule was fixed before training, and it chose step 4,500. The benchmarks were never used to
choose.
- Data. 153,988 training records, with 30,776 for calibration and 123,177 for test (50/10/40).
- Sources. Public datasets and generated decision records.
- Not published. The training files are not part of this release; the SHA-256 of each file is in
training_config.json.
Files
License
Apache-2.0, see LICENSE.
