pplx-decider-flash-v1.2-text-GGUF

Files: pplx-decider-flash-v1.2-{Q8_0,Q4_K_M,IQ4_XS,Q3_K_M}.gguf, plus readout.safetensors, configs, tokenizer and runner/.

GGUF quantizations (Q8_0, Q4_K_M, IQ4_XS, Q3_K_M) of the text backbone of pplx-decider-flash-v1.2. Text-only (frozen copy of the original text release). The vision build (mmproj + image-capable runner) lives at Jakevin/pplx-decider-flash-v1.2-GGUF.

Mixed-precision (D3 sensitivity-allocated) GGUF builds — same size as IQ4_XS / Q4_K_M with lower KL: Jakevin/pplx-decider-flash-v1.2-mixed-GGUF.

Provenance and license

This is a derivative work of perplexity-ai/pplx-decider-flash-v1.2 (revision 6d771ad7b5ecff5e93b8fc7b90d4c5ccc341954d), itself derived from Qwen/Qwen3.5-4B. It is distributed under the same Apache License 2.0; the original LICENSE and NOTICE files are included unchanged. This is not an official Perplexity release.

Modifications from the original checkpoint

• Conversion: llama.cpp convert_hf_to_gguf.py (commit f74e2fa, auto-detected as pplx-decider, arch qwen35) to an intermediate BF16 GGUF (not published), then llama-quantize to Q8_0 / Q4_K_M / Q3_K_M / IQ4_XS (llama.cpp's standard per-tensor K-quant mixes, block sizes 32/256; no imatrix).
• Vision tower dropped: only the text backbone is converted (no mmproj).
• Weight renaming: tensors renamed to llama.cpp GGUF names (blk.N.*, token_embd, …) by the converter.
• Padded lm_head: the converter stores the 255-row readout as a full-vocabulary output.weight that is zero except at the 255 answer-code token ids (adds ~1.27 GB in BF16; quantized together with the file). It is not used by the recommended path.
• Readout kept BF16: the original readout.safetensors is included unchanged and applied externally in fp32.
• Custom non-causal runner required: the GGUF metadata says causal attention and llama.cpp's built-in pplx-decider decision template targets the older v1 single-question prompt; neither matches v1.2. Use runner/run_gguf.py: llama-server --embeddings --pooling none --override-kv qwen35.attention.causal=bool:false -c 512 -ub 512, send the token ids built by decider_common.Encoder (runner/run_gguf.py is the evaluation script used here; it expects llama-server under ./llama.cpp/build/bin and a local copy of the original checkpoint via --ckpt), take the per-token hidden states at the <|fim_suffix|> markers and multiply by the readout. (With causal attention, Q4_K_M had 13 confident flips and KL max 1.43 on the set below.)
• decision_config.json, joint_config.json and the tokenizer files are copied unchanged from the original repo for the runner.

Evaluation (agreement with the original BF16 model)

Reference: the unquantized BF16 weights run through the same non-causal runner (validated against the original PyTorch pplx_decider non-causal hook: 14/14 identical answers, max |Δlogit| 0.06; independent llama.cpp BF16 path: 246/246 identical). Held-out set: 244 requests / 246 questions (model-card example; validation splits of SST-2, MNLI-matched, BoolQ, dair-ai/emotion, STS-B). "Confident flips" = answer changed where the BF16 top-1 margin was > 0.2. KL is on calibrated (T=1.554) probabilities.

Sizes include the padded lm_head. Recommended: Q8_0 (lossless on this set) or Q4_K_M. Small sample; validate on your own task.

An independent BF16 GGUF run matched the original BF16 on all 246 questions (KL max 6e-4). MLX build: Jakevin/pplx-decider-flash-v1.2-mlx.