pplx-decider-v1.1-27b — GGUF

llama.cpp port of perplexity-ai/pplx-decider-v1.1-27b, a decision model: it does not generate text, it answers typed questions about a shared state and returns calibrated probabilities. Same request shape as TypeSafe's Jev (state + noul / choice / score questions).

The upstream checkpoint is bf16 safetensors that needs torch and ~49 GB of weights. This repo makes it runnable under llama.cpp.

Both quants carry the decision head inside the GGUF as cls.output.weight; readout.safetensors is shipped as well so a runtime can load it independently.

Three things to know before deploying this

1. The attention is bidirectional, so there is no prefix to cache. The checkpoint's attention_mode is noncausal_full_attention: 16 of the 64 layers run full attention with the causal mask removed, the other 48 are linear/recurrent and keep theirs. The prompt is ordered State → Question → Options, so the state's representation depends on which question follows it. Nothing can be reused between questions, and the upstream code enforces this by refusing a KV cache outright. Each question is a full forward pass over the whole state. The first full-attention layer is index 3, so the theoretical ceiling for prefix reuse is 3 of 64 layers (~5 %).

This is the same property that makes the model good: because it can re-read the subject after seeing the question, it handles Czech case marking and word order better than causal decision models. Accuracy and throughput are traded against each other here, not independent.

2. Over 8192 tokens it refuses rather than truncates. prepare() raises instead of cutting the record ("no input was truncated"), and the server in runtime/ returns HTTP 400. That is worth preferring over a model that silently answers from a fragment with high confidence.

3. It is a dense 27B. ~25.6 B parameters (after dropping the vision tower), no MoE, all of it active on every pass. Measured ~1.05 ms per token of state on a GB10: 0.56 s at 0.5k tokens, 8.7 s at the 8192-token maximum. It will not fit an 8 GB card in any of these quants.

Quantization

llama-imatrix over 5.0 MB (1707 prompts) rendered with the model's own decision_messages() and chat template from 603 real Czech and English article states, with a mix of noul / choice / score questions and option counts from 2 to 23. Calibration in the production prompt shape, not plain prose.

The quality metric is KL divergence on the decision distribution, not token perplexity. This model has no language head, so llama-perplexity and the generic KL tooling measure something it never does — the reported perplexity during imatrix collection is ~5·10⁵ and is meaningless. The numbers here compare the 255-way option distribution against the bf16 GGUF, on 240 held-out states (a different seed from the calibration corpus).

All ten candidates, measured identically:

A paired bootstrap (10 000 resamples over the same 240 items) on the three best:

Q6_K is therefore not published: it is 1.4 GB larger than Q5_K_M-dyn with no measurable quality advantage on this set. Q8_0 is a real improvement and is published for anyone who wants the headroom.

What did not work

Published recipes are usually a list of things that helped. These are the things that did not, because they cost a day to find out:

• cls.output.weight=f16 is a no-op. llama-quantize leaves the 255×5120 decision head at bf16 on its own. --output-tensor-type does not reach it either — that flag targets output.weight, which this checkpoint does not have.
• Forcing the recurrent gains to f32 makes things worse. ssm_alpha/beta/dt/conv1d = f32 alone raised KL at both Q4 (0.0022 → 0.0024) and Q5 (0.0028 → 0.0041), and doubled the flips. The gain in Q5_K_M-dyn comes from the rest of the layout, chiefly ssm_out and the attention projections at q8_0.
• --token-embedding-type q8_0 changed nothing measurable (0.002188 → 0.002287).
• Lower KL does not mean better decisions. Q6_K + ssm f32 has a lower mean KL than plain Q6_K (0.000453 vs 0.000588) and three times the flips (3 vs 1). Selecting on mean KL alone would have picked the worse model.
• A layout lifted from a different model in the same family (Clef-Flash) did not transfer.

Caveats on these numbers

240 items is a small sample and KL is heavily skewed — the worst 5 % of items carry 45–71 % of the total. That is why the confidence intervals are wide and why differences inside the 2–4·10⁻³ group should not be read as a ranking. The two gaps claimed above are the only ones the bootstrap supports.

Port fidelity

Before any quantization, the bf16 GGUF was checked against the original torch checkpoint on 25 Czech and English items:

The causal control is the proof that llama_set_causal_attn(ctx, false) is doing real work and that llama.cpp's routing matches the model's own hook: causal_attn reaches only set_input_kq_mask, i.e. the full-attention layers, leaving the recurrent ones alone.

All quant numbers above are measured against this bf16 GGUF, so port error (4·10⁻⁴) and quantization error are not mixed together.

Running it

runtime/pplx_bridge.cpp is a small binary that takes token ids on stdin and returns the last token's hidden state; runtime/pplx_api.py puts a Jev-compatible POST /v1/systemone in front of it and applies the readout.

g++ -O2 -std=c++17 -I llama.cpp/include -I llama.cpp/ggml/include -I llama.cpp/src \
    runtime/pplx_bridge.cpp -L <build>/bin -lllama -lggml-base \
    -Wl,--allow-shlib-undefined -Wl,-rpath,<build>/bin -o pplx_bridge

MODEL_DIR=. GGUF=./pplx-decider-v1.1-27b-Q5_K_M-dyn.gguf BRIDGE=./pplx_bridge \
  python3 runtime/pplx_api.py

curl -s localhost:8012/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "Spartu porazila Plzeň 2:1.",
  "questions": {
    "won":   {"type": "noul", "instructions": "Vyhrála Plzeň?"},
    "topic": {"type": "choice", "instructions": "Classify the state.",
              "criteria": {"sport": "sport", "politics": "politics", "other": "anything else"}}
  }}'

Two requirements the bridge has that a normal llama.cpp model does not: causal_attn off, and n_ubatch large enough for the whole prompt — chunking the prefill would cut the bidirectional attention window at every chunk boundary.

How it compares

Measured on identical item sets against hosted Jev and Liquid d1 (via Vercel AI Gateway) and our own Clef-Flash and Rune deployments:

It is the most accurate and by far the best calibrated decision model we have measured, and the only one that gets any of the Czech object-verb-subject items right — "Spartu porazila Plzeň 2:1." / "Vyhrála Plzeň?" is answered 1.00 here and 0.00–0.02 by everything else. It still misses two of the four OVS items, both times confidently.

The cost is throughput: no prefix sharing means a 10-question evaluation over one state costs ten full passes (21.7 s at 2k tokens, versus 2.0 s for Clef-Flash, which answers all ten in one pass). Pick this model when the verdict matters more than the rate.

Licence

Apache 2.0, inherited from the upstream checkpoint. runtime/autojev/ is the upstream prompt and answer code, redistributed under the same licence.