Saina Helm 2 0.8B

A small decision model. Give it some state and any number of typed questions (yes/no, single choice, multiple choice or rating) and it returns a probability for every answer. No text is generated.

Options are supplied at request time, so one model handles intent routing, triage, tool selection, review checklists and similar decisions without retraining.

It scores 39.26 on the public part of Decision Index 0.3, the highest of any model under 1B on the board, and has the highest tool-use (API-Bank, BFCL, When2Call) and RAGTruth hallucination scores of any model under 1B. See Evaluation.

New in Helm 2: every question in a request is answered from one pass over the shared state, instead of one pass per question, and the readout gains small routed expert heads. Previous model: saina-helm-0.8b.

Intended use

Decisions where a wrong answer is recoverable or reviewed before it takes effect: intent routing, triage, tool shortlisting, review checklists. Where a wrong answer has a cost, run with mode='decision' so low-confidence questions escalate instead of guessing. Validate on your own labelled data, for your own label space, before use.

Out of scope

• Generating text of any kind; the model only scores the options you supply.
• Final, unreviewed decisions with material consequences for a person (credit, employment, medical, legal), even at high confidence.
• Classifying people: personal attributes, demographics, or anything used to profile individuals.
• Safety filtering, harm detection or content moderation; not trained or evaluated for it.
• Non-English input. Training data is mostly English with limited non-English material, every reported evaluation is English, and other languages are untested.
• Decisions taken directly on untrusted text. The model scores whatever state it is given, so injected content can steer its answers; bound the authority you grant the output.

Usage

pip install "saina[local]>=0.3.2"

from saina.transformers import HelmModel

helm = HelmModel.from_pretrained('run-saina/saina-helm-2-0.8b', revision='<commit>', device='cuda:0')

answers = helm.ask(
    model='saina-helm-2-0.8b',
    state='Order #4417: parcel arrived crushed, customer wants their money back. Third complaint this month.',
    questions={
        'refund': {'type': 'yes_no', 'question': 'Is a refund requested?'},
        'team': {'type': 'single_choice', 'question': 'Which team should handle this?',
                 'options': {'billing': 'Payments and refunds', 'shipping': 'Delivery and carriers', 'tech': None}},
        'issues': {'type': 'multi_choice', 'question': 'Which issues apply?',
                   'options': {'damaged': 'Item damaged', 'late': 'Late delivery', 'repeat': 'Repeat complainer'}},
        'urgency': {'type': 'rating', 'question': 'How urgent is this?',
                    'levels': ['Low', 'Medium', 'High', 'Critical']},
    },
)

Pin revision to a commit. The saina package builds the exact prompt the model was trained on and applies the trained head; scores depend on that format, so use it rather than a custom prompt or generate(). No remote code is executed. Helm 2 needs saina 0.3 or newer; older versions refuse this checkpoint.

The same package runs an authenticated HTTP server (pip install "saina[local,server]", see run-saina/saina).

Question types

Decision mode. Pass mode='decision' with a threshold (default 0.8) and, for yes_no / single_choice, an optional min_margin; both can be overridden per question. Answers then include selected / selection / selections / level, set to None with a reason (below_threshold, below_margin, tie) when the model is not confident enough, so you can abstain and escalate instead of guessing.

Context length

The checkpoint was trained with a 262,144-token limit (the base model's), which saina uses by default. A request is one forward pass over the whole input, so GPU memory grows with input length: about 3.2 GiB at 15K tokens and 7.9 GiB at 58K on an RTX 3090 (measured), roughly 0.11 GiB per 1K tokens, so the full 262K needs on the order of 30 GiB. Pass max_length= to serve a limit that fits your hardware. Longer inputs are rejected, never truncated. Installing flash-linear-attention and causal-conv1d enables faster kernels.

Evaluation

This model scores 39.26 on the public part of Decision Index 0.3, the highest among models under 1B parameters on the board, ahead of vLLM-SR d3-lite (36.93) and Crivo-150M (36.39). OneJev1.5-0.8B scores 42.71 but serves about 1.1B parameters, so it is not counted here. Larger models on the board score higher. The board's official Full score, which adds the maintainers' private tests, is 20.82 for the earlier revision 06f2fec2 (public part 30.47), fifth among models under 1B; it is not yet available for 72349026. Board scores are from the leaderboard data of 2026-10-16.

Run against the Decision Index 0.3 benchmark suite with the reproducible harness in run-saina/saina-decision-index: all 43 benchmarks, 140,178 requests, every one answered, on revision 72349026.

By area (chance-corrected, 0 = random, 100 = perfect):

Per benchmark, each scored on its complete set, against six sub-1B models from the leaderboard's saved public results of 2026-10-10 (jiwo 0.8B, Dinah-0, Sifr 0.8B, Bekko 400M, OneJev 0.8B, Eos 0.8B). Rank is among these seven.

Helm 2 is strongest on tool use and structured decisions (BFCL, API-Bank, When2Call, ContractNLI, RAGTruth); its BFCL (92.5), API-Bank (81.7), When2Call (83.8) and RAGTruth (77.5) scores are the highest of any sub-1B model on the board (per-benchmark results of 2026-10-16). It is weak on knowledge and reasoning (GPQA Diamond, CRUXEval, BBH) and on retrieval ranking (BRIGHT, ToolRet), where it places fourth to seventh of seven. On PhishNChips it scores close to chance (50.9), and on Home appliances it gets only 6.8% of cases fully right, far behind Crivo-150M (79.5).

This run sends every benchmark through the same request rendering API callers get from /v1/systemone; the results above are what users get.

At 81.7% API-Bank accuracy roughly one tool selection in five is wrong: treat tool choice as a ranked shortlist to review, not a final answer.

Training

Official test records were excluded from training by an exact-match audit; near-duplicates were not checked.

Limitations

Scope:

• Fairness across dialects, demographics and languages has not been evaluated.
• Domains and label spaces far from the training mix are untested; the results above do not predict them.
• Results are sensitive to how questions and options are phrased. Fix a wording, keep it stable, and revalidate when it changes.

Mechanics:

• Probabilities are not calibrated; set thresholds on your own data.
• Helm 2 runs with the saina Python runtime, and with llama.cpp (GGUF) and MLX through the scorer each ships. Its head is not part of the GGUF, so stock Ollama, llama-server and llama-cli can't serve it.

Feedback

Issue reports and results from your own tasks are welcome on run-saina/saina.

License

The weights in this repository are released under Apache-2.0. Training data sources keep their own terms, and some are share-alike or have unclear terms; no representation is made that the weights are cleared for commercial use. Review your use accordingly.