Sarvam-Prism

Sarvam-Prism is a decision model built on Sarvam-105B. You give it a context and one or more questions, each with a fixed set of possible answers. It returns a probability for every answer in a single forward pass. It does not generate text, so the answer is always one of your options.

Release blog: sarvam.ai/blogs/sarvam-prism: evaluation results, comparisons with other decision models, calibration, and how the model was built and made fast.

API: Sarvam-Prism is also available as a hosted API from Sarvam, with no GPU needed: see the API documentation.

Requirements

• One NVIDIA GPU with FP8 support and about 120 GB of free memory, such as a B200 or H200. Tested on B200.
• Python 3.10+ and a C compiler (Triton builds its kernels on first use).
• About 106 GB of disk for the weights.

Quick start

1. Download the model and code:

pip install -U huggingface_hub
huggingface-cli download sarvamai/prism --local-dir sarvam-prism
cd sarvam-prism

2. Install the dependencies:

pip install -r requirements.txt

3. Run the example:

python example.py .

Loading takes about two minutes on a B200 (Triton compiles its kernels the first time, then CUDA graphs are captured). It prints the answers to the request below.

Use it from Python (run from the sarvam-prism folder, or put it on your PYTHONPATH):

from prism import Prism

model = Prism.from_pretrained(".")   # the downloaded folder; loads on cuda:0

answers = model.decide(
    state={"ticket": "My card was charged twice for the same order and I still haven't got a refund."},
    questions={
        "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
                 "criteria": {"billing": "payments, charges and refunds",
                              "tech": "app or website problems",
                              "delivery": "shipping and order status"}},
        "urgent": {"type": "yes_no", "instructions": "Does this need a reply within the hour?"},
        "anger": {"type": "score", "instructions": "How upset is the customer?",
                  "criteria": ["calm", "annoyed", "angry", "furious"]},
    },
)

The output has one entry per question:

{
  "team":   {"type": "choice", "choice": "billing",
             "probabilities": {"billing": 0.986, "delivery": 0.006, "tech": 0.007}},
  "urgent": {"type": "yes_no", "yes": 0.45},
  "anger":  {"type": "score", "score": 1.86, "level": "2",
             "probabilities": {"0": 0.032, "1": 0.267, "2": 0.510, "3": 0.191}}
}

Prism.from_pretrained("sarvamai/prism") also accepts the repo id and downloads the weights into the Hugging Face cache on first use (you still need prism.py from this repository).

Load the model with prism.py as shown. The weights are stored in a custom FP8 layout for this code, so they cannot be loaded with AutoModelForCausalLM or served by general-purpose inference servers.

Speed

Median 44 ms per request (95th percentile 420 ms) on one NVIDIA B200, measured on 1,000 random requests from the public Decision Index benchmark, sent one at a time after loading. All questions in a request are scored together.

Request format

• state: the context to decide about. Any text or JSON-serialisable value; JSON is rendered compactly.
• questions: a map from your question names to questions. Each question has:
  • type: one of the three below.
  • instructions: what to decide, in plain language.
  • criteria: the possible answers.

All questions in a request are scored together, in batches of up to batch_size (default 16, set in from_pretrained), and each question is scored independently of the others. Choice options are scored in a fixed order (sorted by key), so the same options always give the same answer, whatever order you send them in. The context limit is 131,072 tokens per question.

Loading captures CUDA graphs for common batch sizes and lengths, which takes a few minutes and makes most requests much faster; pass cuda_graphs=False to skip it.

How it works

Each question becomes one sequence: the state, the instructions, and every option, followed by an answer marker. Every option starts at the same position and cannot see the other options; only the answer marker sees all of them. A small decision head reads the hidden state at the end of each option and at the answer marker, and turns them into one probability per option. prism.py contains the full inference code: the input encoding, the attention mask, the decision head, two small changes to the base model's attention for this mask, and the FP8 kernels.

The weights are stored in FP8 (float8_e4m3fn) with one scale per output row; activations are scaled per token at run time. The 128 routed experts of each layer run as two grouped FP8 matrix multiplications (a Triton kernel), and the other projections use torch._scaled_mm. Embeddings and norms are bf16, and the MoE router is fp32.

Files

License

Apache 2.0 (see LICENSE).