pplx-decider-flash-v1.2

pplx-decider-flash-v1.2 is the small version of pplx-decider-v1.2: the same joint request format, the same API and the same text and image support, at about one seventh of the size. It is built on Qwen3.5-4B, distilled from the 27B decision models on their training requests (the student learns the teachers' full probability distributions, not just their answers). It combines several training runs through model souping (weight averaging).

Use it when latency or cost matters more than the last points of accuracy. On our evaluations it keeps most of the 27B model's Decision Index and vision scores (see below). Distillation also improves results on decision tasks held out from all training.

GGUF files

Stock llama-server does not implement this checkpoint's protocol v3; its /v1/systemone uses protocol v1. A compatible runtime must:

• build the joint inline-readout prompt (protocol v3; see source/src/pplx_decider/protocol.py in the base repo);
• evaluate the whole prompt, including image embeddings, in one microbatch; full-attention layers are non-causal, linear-attention layers stay causal;
• read the logits at each <|fim_suffix|> marker for that question's options using pplx_decider.token_ids, divide by pplx_decider.temperature and apply a softmax over those options.

The readout is stored as output.weight, zero outside the code tokens. qwen35.attention.causal is false. The other protocol data are in pplx_decider.* keys: protocol_version, template, marker, temperature, codes, token_ids, escape_tokens, max_pixels.

On Apple Metal, Q8_0 matched the float32 reference's highest-probability option on all 13 golden questions (text and images), each run separately. The largest probability difference was 0.016.

The evaluation results, Python usage, requirements, limits and file descriptions below are from the base model and describe that release, not these GGUF files.

Evaluation

The overall score uses the suite's weighting, not a simple category average. Joint requests: all of a request's questions share one sequence. One question per request: every question is sent alone with its state. Unlike the 27B model, this model is more accurate with one question per request on most multi-question tasks (see When to group questions); the two modes give the same score on the other benchmarks.

Usage

Requirements: Python 3.12+ and a CUDA GPU with room for about 9 GiB of BF16 weights plus working memory. The repository is private: authenticate with a Hugging Face token that has access to it. Download the repository and install requirements.txt. That file pins torch, transformers, safetensors, pillow, torchvision and huggingface-hub. The fast path also needs requirements-fast.txt.

import sys
from pathlib import Path
from huggingface_hub import snapshot_download

checkpoint = Path(snapshot_download("perplexity-ai/pplx-decider-flash-v1.2"))
sys.path.insert(0, str(checkpoint / "source" / "src"))
from pplx_decider import Decider

decider = Decider.from_pretrained(checkpoint, device="cuda")
response = decider.decide({
    "state": "My integration keeps failing with a 401 since we rotated our API key. Please help.",
    "questions": {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "billing": "Charges and refunds",
                "support": "Technical integration errors",
                "sales": "Questions about buying a product",
            },
        },
        "urgent": {"type": "noul", "instructions": "Is the customer blocked right now?"},
        "frustration": {"type": "score", "criteria": ["calm", "mildly annoyed", "frustrated", "angry"]},
    },
})
print(response["answers"]["team"]["choice"], response["answers"]["urgent"]["noul"])

decide accepts one request and returns one response. Given a list of requests, it returns a list of responses in the same order. The source can also be installed as a package: pip install ./source (or pip install "./source[fast]").

Request schema

{
  "state": str | dict | list,            # the context to classify; non-strings are serialized as JSON
  "questions": {                         # 1 or more questions, answered from the same state
    "<id>": {
      "type": "choice" | "noul" | "score",
      "instructions": str | None,        # optional; default "Choose the best matching option."
      "criteria": ...,                   # see below
    },
  },
  "images": [path | "data:image/...;base64,..." | PIL.Image],   # optional
}

The options are shown to the model in the order you give them, under the answer codes A, B, ... Each response looks like {"answers": {"<id>": {...}}, "usage": {"input_tokens": int, "output_tokens": 0, "sequences": int}}. input_tokens counts every non-padding position the request used, including the template and image positions. sequences is 1 unless the request had to be split. Requests that do not follow the schema raise pplx_decider.RequestError before any computation.

decider.logits(requests) returns the raw readout logits of each question's valid options, before temperature. For ranking, use the logit margin true - false of a noul question: unlike probabilities, it does not saturate into ties.

When to group questions

All questions in a request share one sequence, and every question attends to the state and to every other question. For this model, one question per request is the safer default. Grouping helps only where the questions are tightly coupled facets of one decision, and it hurts more than for the 27B model elsewhere:

(Decision Index task scores, points.) Our recommendations:

• Send one question per request by default, and score independent candidates (documents, tools, calls) in separate requests. The fast path packs many small requests efficiently, so separate requests stay cheap.
• Group only multi-label "select all that apply" decisions or steps of one routing decision, and check on your task.
• Question IDs are not shown to the model, but question order is. Put the context the questions need in the state.

Temperature and calibration

decision_config.json stores the temperature fitted on held-out calibration data (1.554 for this checkpoint). decide applies it exactly once, then normalizes a softmax over each question's valid options only. decide(..., temperature=t) overrides it. When you use logits directly, divide by the saved temperature before the softmax, and do not include the unused codes. It was fitted in the joint format, on data separate from the development set. On the development set (1,497 questions) it gives accuracy 88.8%, ECE 0.021 and Brier 0.163 (without temperature: ECE 0.057, Brier 0.173).

Images

Images go in images. They are placed before the text of the request, and every question can see them. Each image is resized (aspect ratio kept) to at most max_pixels pixels: 1,638,400 (1.6 MP), as saved in decision_config.json. Smaller images keep their size (minimum 65,536 pixels). A 1.6 MP image uses about 1,600 sequence positions, so several large images count toward the length limit. Pass max_pixels= to from_pretrained to change the cap. The checkpoint was trained and evaluated at its saved value.

Fast path

Decider.from_pretrained(..., fast=True) packs whole requests into padding-free sequences, compiles the decoder layers and replays CUDA graphs, exactly as for pplx-decider-v1.2 (see that model card for the details and options). It needs requirements-fast.txt. Results match the reference path up to BF16 noise.

Limits

• Options: 1 to 255 per question (2 to 255 for score).
• Length: max_length defaults to 262,144 tokens per sequence, the backbone's position limit. Training used sequences up to about 32k tokens. Longer requests still run as one sequence, but we have only spot-checked them: retrieving one fact from a long log works up to about 243k tokens (P(correct) above 0.999), with peak GPU memory of about 39 GiB (reference path) or 54 GiB (fast path), and about 1 minute (reference) or 20 seconds (fast) per 243k-token request on one GB200. Validate beyond 32k on your own task, or pass max_length=32768. When a request exceeds max_length, its questions are split in halves, recursively, and every chunk keeps the full state and images; a single question that does not fit raises pplx_decider.LengthError. Inputs are never truncated.
• Questions per request: no fixed cap; the length limit is the only constraint (but see When to group questions).

Checkpoint format and serving

This is the native decision-checkpoint layout: a Qwen3_5Model backbone plus readout.safetensors, a BF16 [255, 2560] decision head. The Transformers class name does not change the base model identity, Qwen3.5-4B. joint_config.json declares the request protocol (version: 3, joint-question-inline-readout-v1, marker <|fim_suffix|>).

• Full-attention layers are noncausal over the whole request. The linear-attention layers keep their native causal behavior. Default causal inference does not reproduce this checkpoint.
• The readout is applied at each <|fim_suffix|> marker, one per question, not at the last token. Literal special tokens in user content are neutralized before tokenization, so user text cannot create markers.
• This artifact has a separate readout, not a full-vocabulary lm_head. A serving system needs both the noncausal attention and the per-marker readout. Readout row i corresponds to vocabulary row decision_config.json["token_ids"][i].

release-manifest.json records file checksums.

Troubleshooting

• ImportError: ... flash-linear-attention: install requirements-fast.txt, or use fast=False.
• Slow first call with fast=True: torch.compile and CUDA-graph capture happen on first use of each bucket. Call decider.warmup() at startup. Pass buckets=(32768,) for a single shape, or compile=None to skip compiling.
• Probabilities differ slightly between paths or batch compositions: expected in BF16 (mean |Δp| about 0.002, with rare argmax flips on near-ties). Use fast=False and one request per call for bit-stable results.
• LengthError: one question with the state and images exceeds max_length. Shorten the state, lower max_pixels, or raise max_length up to the backbone's 262,144 (beyond about 32k tokens, the training length, the probabilities are less validated).
• Questions influence each other: by design, see When to group questions. Use one request per question for fully independent answers.
• Out of memory: lower batch_tokens (reference path) or use smaller buckets / token_budget (fast path).
• Different answers from v1.1-style prompts: this model expects the joint format built by pplx_decider. Do not reuse the v1.1 single-question prompt.

Acknowledgment

A large part of the gains are thanks to the tasksource data. If you also use this data, consider citing the corresponding article

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}