pplx-decider-v1.3 is the next version of pplx-decider-v1.2, a multimodal classification model on Qwen3.8-27B. It comes with improved performance and robustness across decision tasks, and a new extraction capability: Locate, which extracts verbatim text spans from the state with character offsets and confidence scores.
If you are interested in the performance of the models, how we train it and evaluate it, as well as use cases examples, you can read the blogpost here: https://www.perplexity.ai/hub/blog/perplexity-decider
Requirements: Python 3.12+ and a CUDA GPU with room for about 49 GiB of BF16 weights plus working memory.
The repository is private: authenticate with a Hugging Face token that has access to it.
Download the repository and install requirements.txt (torch, transformers, safetensors, pillow, torchvision, huggingface-hub).
import sys
from pathlib import Path
from huggingface_hub import snapshot_download
checkpoint = Path(snapshot_download("perplexity-ai/pplx-decider-v1.3"))
sys.path.insert(0, str(checkpoint / "source" / "src"))
from pplx_decider import Decider
decider = Decider.from_pretrained(checkpoint, device="cuda")
response = decider.decide({
"state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this request?",
"criteria": {
"billing": "Charges and refunds",
"support": "Technical integration errors",
"sales": "Questions about buying a product",
},
},
"urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
"frustration": {"type": "score", "criteria": ["calm", "mildly annoyed", "frustrated", "angry"]},
},
})
print(response["answers"]["team"]["choice"], response["answers"]["urgent"]["noul"])
decide accepts one request and returns one response. Given a list of requests, it returns a list of responses in
the same order. The source can also be installed as a package: pip install ./source.
{
"state": str | dict | list, # the context to classify; non-strings are serialized as JSON
"questions": { # 1 or more questions, answered from the same state
"<id>": {
"type": "choice" | "noul" | "score" | "locate",
"instructions": str | None, # optional; for Locate, refines/replaces the question key
"criteria": ..., # see below
},
},
"images": [path | "data:image/...;base64,..." | PIL.Image], # optional
}
| Type | criteria |
Answer |
|---|---|---|
choice |
{option_name: description or None}, 1 to 255 options |
choice, probabilities (per option), confidence |
noul |
optional {"true": description, "false": description} |
noul = P(true) |
score |
list of 2 to 255 level descriptions, lowest first | score (expected level index), probabilities, legend, confidence |
locate |
not used; question key is the label, optionally replaced by instructions |
spans: start, end, text, score |
The options are shown to the model in the order you give them, under the answer codes A, B, ... Each response
looks like {"answers": {"<id>": {...}}, "usage": {"input_tokens": int, "output_tokens": 0, "sequences": int}}.
Requests that do not follow the schema raise pplx_decider.RequestError before any computation.
decider.logits(requests) returns the raw readout logits of each question's valid options, before temperature.
For ranking, use the logit margin true - false of a noul question: unlike probabilities, it does not saturate
into ties.
For decision questions, a request can carry any number of questions about the same state. By default (mode="separate") each question is answered
in its own sequence, made of the state (and images) and that question only: questions do not see each other, and an
answer does not change when you add, remove or reorder the other questions. This is the computation all the scores above
were measured with. Put everything a question needs in the state, not in another question.
Decider.from_pretrained(..., mode="isolated") instead encodes a whole request as one sequence with the training-time
layout: the state is read once and each question attends to the state and to itself. It is cheaper when many questions
share a long state, but the state also attends to the questions, so answers can shift slightly with the set of questions
(on our samples, 1.5-2.2% of top answers differ from separate). Use separate unless the cost of re-reading the state
matters.
decision_config.json stores the temperature fitted on held-out calibration data. decide applies it exactly once,
then normalizes a softmax over each question's valid options only. decide(..., temperature=t) overrides it. When you
use logits directly, divide by the saved temperature before the softmax, and do not include the unused codes.
If your application relies on probability values (rather than rankings or the top answer), calibrate on a sample of
your own traffic: a single temperature cannot be right for every task, and per-task thresholds are usually better.
Images go in images. They are placed before the text of the request, and every question can see them. Each image is
resized (aspect ratio kept) to at most max_pixels pixels: 1,638,400 (1.6 MP), as saved in
decision_config.json. Smaller images keep their size. A 1.6 MP image uses about 1,600 sequence positions, so several
large images count toward the length limit. The checkpoint was trained and evaluated at its saved value.
Decider.from_pretrained(..., fast=True) (requires requirements-fast.txt) packs many sequences per forward pass and
compiles the decoder with torch.compile. It supports both modes on all three model sizes:
mode="separate" also replays CUDA graphs. On our validation sample it was about 1.7-1.9x faster than the default
padded batches, with 99.6-99.7% top-answer agreement with the reference path.mode="isolated" reads each request's state once and starts every question from it, as the reference isolated path
does (no CUDA graphs: their shapes depend on the request). On our validation sample it was 9-18x faster than the
reference isolated path and faster than the separate fast path, with 99.4-99.8% top-answer agreement with the
reference isolated path.Differences are BF16 batching noise (mean |Δp| about 0.002, about 0.006 for the mixture-of-experts model). Call
decider.warmup() at startup to compile ahead of time. The default path with one request per call reproduces our
evaluation implementation to within 3e-7.
Use "type": "locate" to extract matching text spans from the state. The question key is the default label:
"person": {"type": "locate"} extracts people. Optional instructions refines the request by replacing that label,
for example "person": {"type": "locate", "instructions": "The person who signed the contract"}.
Missing, null or blank instructions uses the key. Locate questions can be mixed with decision questions.
The adapter is loaded automatically on the first Locate request.
decider = Decider.from_pretrained(checkpoint, device="cuda")
response = decider.decide({
"state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
"questions": {
"urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
"error": {"type": "locate", "instructions": "Error codes"},
},
})
print(response["answers"]["urgent"]["noul"])
for span in response["answers"]["error"]["spans"]:
print(span["start"], span["end"], span["text"], span["score"])
Locate requires a nonempty text state, no images and at most 128 questions per request.
Each span has character offsets start and end (exclusive), its text, and a score;
state[start:end] == text. No matches returns an empty spans list.
Locate questions run together, separately from decisions, and can influence each other through the shared state.
Requests mixing Locate and decision questions require separate forward passes for each type, adding latency
compared with requests containing only one type. Splitting the same work into two decide() calls does not
remove these forward passes.
They use the same max_length setting (default 32,768 tokens): questions are split only when needed, preserving
the full state. If a single question plus the state cannot fit, LengthError is raised.
fast=True and locate=False disable Locate.
Calls to decide(), logits(), reference_logits() and warmup() on the same Decider instance
are serialized by an internal lock, including adapter loading. Concurrent callers are safe through these methods;
use separate instances for parallel inference.
score). The answer codes are single tokens, and the readout
head has 255 rows.This is the native decision-checkpoint layout: the backbone plus readout.safetensors, a BF16 [255, hidden_size]
decision head. joint_config.json declares the request protocol (version: 4, joint-chunked-readout-v1, marker
<|fim_suffix|>, separator <|fim_middle|>).
<|fim_suffix|> marker, one per question, not at the last token. Literal special
tokens in user content are neutralized before tokenization, so user text cannot create markers.lm_head. Readout row i corresponds to vocabulary
row decision_config.json["token_ids"][i].release-manifest.json records file checksums.
LengthError: one question with the state and images exceeds max_length. Shorten the state or lower
max_pixels.batch_tokens.pplx_decider. Do
not reuse earlier prompt builders.A large part of the gains are thanks to the tasksource data. If you also use this data, consider citing the corresponding article
@inproceedings{sileo-2024-tasksource,
title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
author = "Sileo, Damien",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
month = may,
year = "2024",
address = "Torino, Italia",
publisher = "ELRA and ICCL",
url = "https://aclanthology.org/2024.lrec-main.1361",
pages = "15655--15684",
}
model-00004-of-00011.safetensors 5.0 GBmodel-00007-of-00011.safetensors 5.0 GBmodel-00010-of-00011.safetensors 5.0 GBmodel-00002-of-00011.safetensors 5.0 GBmodel-00008-of-00011.safetensors 5.0 GBmodel-00009-of-00011.safetensors 4.9 GBmodel-00006-of-00011.safetensors 4.9 GBmodel-00005-of-00011.safetensors 4.9 GBmodel-00003-of-00011.safetensors 4.9 GBmodel-00001-of-00011.safetensors 4.8 GBmodel-00011-of-00011.safetensors 2.8 GBlocate-span/adapter.safetensors 1.9 GBlocate-span/head.safetensors 13 MBreadout.safetensors 3 MBQwen/Qwen3.8-27B