pplx-decider-flash-v1.2-mixed-GGUF

Sensitivity-allocated mixed-precision GGUF builds of the text backbone of pplx-decider-flash-v1.2. Text-only. Standalone repo: the GGUF files, readout.safetensors, configs, tokenizer and runner/ are all here. Uniform llama.cpp quants (Q8_0 / Q4_K_M / IQ4_XS / Q3_K_M) are in Jakevin/pplx-decider-flash-v1.2-GGUF.

Provenance and license

This is a derivative work of perplexity-ai/pplx-decider-flash-v1.2 (revision 6d771ad7b5ecff5e93b8fc7b90d4c5ccc341954d), itself derived from Qwen/Qwen3.5-4B. It is distributed under the same Apache License 2.0; the original LICENSE and NOTICE files are included unchanged. This is not an official Perplexity release.

Modifications from the original checkpoint

• Conversion: llama.cpp convert_hf_to_gguf.py (commit f74e2fa, arch qwen35) to an intermediate BF16 GGUF (not published), then llama-quantize --tensor-type-file with an explicit type for every weight tensor (no imatrix).
• Mixed precision (D3 allocation, see below): each layer group gets its own llama.cpp type among Q2_K / Q3_K / Q4_K / Q5_K / Q6_K. No ternary types (TQ1_0 / TQ2_0) are used. Norms, conv1d, ssm_a, ssm_dt stay F32.
• Vision tower dropped: only the text backbone is converted (no mmproj).
• Weight renaming: tensors renamed to llama.cpp GGUF names (blk.N.*, token_embd, …) by the converter.
• Padded lm_head: the converter stores the 255-row readout as a full-vocabulary output.weight that is zero except at the 255 answer-code token ids; it is kept at Q6_K (the same as the uniform quants, so sizes are comparable) and is not used by the runner.
• Readout kept BF16: the original readout.safetensors is included unchanged and applied externally in fp32.
• Custom non-causal runner required: the GGUF metadata says causal attention, and llama.cpp's built-in pplx-decider template targets the v1 single-question prompt; neither matches v1.2. See Usage.
• decision_config.json, joint_config.json and the tokenizer files are copied unchanged from the original repo.

How the bits were allocated (D3)

The model was split into 97 groups: the token embedding, and in each of the 32 layers the token mixer (Gated-DeltaNet attn_qkv / attn_gate / ssm_alpha / ssm_beta / ssm_out, or attention attn_q / k / v / output), the MLP ffn_gate + ffn_up, and ffn_down. Each group was quantized alone (everything else BF16), and the KL divergence of the calibrated decision probabilities vs BF16 was measured at every answer marker (MLX, round-to-nearest formats from ternary to 8-bit). A size-constrained knapsack then picked one llama.cpp type per group, minimising the summed KL at the target file size. Q2_K/Q3_K/Q4_K costs use the KL measured at 2/3/4 bits. Q5_K/Q6_K were not measured directly; their KL is extrapolated from the 4-bit value. The sensitive parts get more bits: the mixers of early layers and of the last layer, and some MLP-down projections. Layers ~20–29 are the least sensitive and get Q2_K/Q3_K.

Bit mix (share of quantized parameters, including the embedding):

• 2.67GiB: Q2_K 6.9%, Q3_K 38.3% (incl. embedding), Q4_K 10.0%, Q5_K 42.9%, Q6_K 1.9%.
• 2.85GiB: Q2_K 0.6%, Q3_K 25.3%, Q4_K 21.3% (incl. embedding), Q5_K 47.1%, Q6_K 5.7%.

Calibration: 54 examples from the train splits of SST-2, MNLI, BoolQ, dair-ai/emotion and STS-B, deduplicated against the 246 evaluation questions (no overlap). No model-card examples and no validation or test data were used.

Evaluation (agreement with the original BF16 model)

Reference: the unquantized BF16 weights run through the same non-causal protocol (validated against the original PyTorch pplx_decider non-causal hook). Held-out set: 244 requests / 246 questions (model-card examples; validation splits of SST-2, MNLI-matched, BoolQ, dair-ai/emotion, STS-B). "Confident flips" = answer changed where the BF16 top-1 margin was > 0.2. KL is on calibrated (T=1.554) probabilities. All GGUF rows were run with the same runner on an Apple M4.

Sizes include the padded Q6_K lm_head (0.49 GiB) for every row.

Noise: this is a small set. One question is ≈0.4 points on the full set and ≈0.5 points on the choice+noul subset, so differences of ±1–2 questions are within noise. For example, both D3 builds are 1 question behind Q4_K_M on choice+noul and 3 questions ahead on the full set. KL and confident flips are the more stable signals: at the same size, the D3 builds have 3–4× lower mean KL than Q4_K_M / IQ4_XS. Validate on your own task.

Usage

Build llama.cpp (tested at commit f74e2fa), then start the server with non-causal attention and per-token embeddings:

llama-server -m pplx-decider-flash-v1.2-D3-mixed-2.67GiB.gguf --embeddings --pooling none \
  --override-kv qwen35.attention.causal=bool:false -c 4096 -b 4096 -ub 4096 -np 1 -ngl 99 --port 8097

Encode the request with runner/decider_common.py, send the token ids, take the hidden states at the <|fim_suffix|> markers and multiply them by the readout:

import json, sys, urllib.request, numpy as np
from huggingface_hub import snapshot_download
from safetensors.torch import load_file   # readout is BF16 (needs torch)
repo = snapshot_download("Jakevin/pplx-decider-flash-v1.2-mixed-GGUF", allow_patterns=["runner/*", "*.json", "*.jinja", "readout.safetensors"])
sys.path.insert(0, f"{repo}/runner")
from decider_common import Encoder, answers
enc = Encoder(repo); W = list(load_file(f"{repo}/readout.safetensors").values())[0].float().numpy()
req = {"state": "My integration keeps failing with a 401 since we rotated our API key.",
       "questions": {"team": {"type": "choice", "criteria": {"billing": None, "support": None, "sales": None}}}}
ids, markers, counts = enc.encode(req)
r = urllib.request.Request("http://127.0.0.1:8097/embeddings", data=json.dumps({"content": ids}).encode(),
                           headers={"Content-Type": "application/json"})
emb = json.loads(urllib.request.urlopen(r).read())[0]["embedding"]
lg = (np.asarray([emb[m] for m in markers], dtype=np.float32) @ W.T).tolist()
print(answers(req, [row[:c] for row, c in zip(lg, counts)], enc.temperature))

runner/run_gguf.py is the evaluation script behind the table. It starts llama-server itself (expected under ./llama.cpp/build/bin); pass --ckpt <this repo dir> for the tokenizer and readout. It also needs an eval_set.json, which is not included. Do not use llama-cli or llama.cpp's built-in decision template: they use causal attention and the v1 prompt.

MLX builds (uniform and D3 mixed): Jakevin/pplx-decider-flash-v1.2-mlx.