pplx-decider-flash-v1.2-GGUF

Files: pplx-decider-flash-v1.2-{Q8_0,Q4_K_M,IQ4_XS,Q3_K_M}.gguf, mmproj-pplx-decider-flash-v1.2-{F16,Q8_0}.gguf, plus readout.safetensors, configs, tokenizer and runner/.

GGUF quantizations (Q8_0, Q4_K_M, IQ4_XS, Q3_K_M) of the text backbone of pplx-decider-flash-v1.2, plus a vision projector (mmproj-*.gguf) for image inputs. A text-only snapshot is at Jakevin/pplx-decider-flash-v1.2-text-GGUF.

Mixed-precision (D3 sensitivity-allocated) GGUF builds — same size as IQ4_XS / Q4_K_M with lower KL: Jakevin/pplx-decider-flash-v1.2-mixed-GGUF.

Provenance and license

This is a derivative work of perplexity-ai/pplx-decider-flash-v1.2 (revision 6d771ad7b5ecff5e93b8fc7b90d4c5ccc341954d), itself derived from Qwen/Qwen3.5-4B. It is distributed under the same Apache License 2.0; the original LICENSE and NOTICE files are included unchanged. This is not an official Perplexity release.

Modifications from the original checkpoint

• Conversion: llama.cpp convert_hf_to_gguf.py (commit f74e2fa, auto-detected as pplx-decider, arch qwen35) to an intermediate BF16 GGUF (not published), then llama-quantize to Q8_0 / Q4_K_M / Q3_K_M / IQ4_XS (llama.cpp's standard per-tensor K-quant mixes, block sizes 32/256; no imatrix).
• Vision tower: converted separately into mmproj-*.gguf (see "Vision (mmproj)" below); the LLM GGUFs are text backbone only.
• Weight renaming: tensors renamed to llama.cpp GGUF names (blk.N.*, token_embd, …) by the converter.
• Padded lm_head: the converter stores the 255-row readout as a full-vocabulary output.weight that is zero except at the 255 answer-code token ids (adds ~1.27 GB in BF16; quantized together with the file). It is not used by the recommended path.
• Readout kept BF16: the original readout.safetensors is included unchanged and applied externally in fp32.
• Custom non-causal runner required: the GGUF metadata says causal attention and llama.cpp's built-in pplx-decider decision template targets the older v1 single-question prompt; neither matches v1.2. Use runner/run_gguf.py: llama-server --embeddings --pooling none --override-kv qwen35.attention.causal=bool:false -c 512 -ub 512, send the token ids built by decider_common.Encoder (runner/run_gguf.py is the evaluation script used here; it expects llama-server under ./llama.cpp/build/bin and a local copy of the original checkpoint via --ckpt), take the per-token hidden states at the <|fim_suffix|> markers and multiply by the readout. (With causal attention, Q4_K_M had 13 confident flips and KL max 1.43 on the set below.)
• decision_config.json, joint_config.json and the tokenizer files are copied unchanged from the original repo for the runner.

Evaluation (agreement with the original BF16 model)

Reference: the unquantized BF16 weights run through the same non-causal runner (validated against the original PyTorch pplx_decider non-causal hook: 14/14 identical answers, max |Δlogit| 0.06; independent llama.cpp BF16 path: 246/246 identical). Held-out set: 244 requests / 246 questions (model-card example; validation splits of SST-2, MNLI-matched, BoolQ, dair-ai/emotion, STS-B). "Confident flips" = answer changed where the BF16 top-1 margin was > 0.2. KL is on calibrated (T=1.554) probabilities.

Sizes include the padded lm_head. Recommended: Q8_0 (lossless on this set) or Q4_K_M. Small sample; validate on your own task.

An independent BF16 GGUF run matched the original BF16 on all 246 questions (KL max 6e-4). MLX build: Jakevin/pplx-decider-flash-v1.2-mlx.

Vision (mmproj)

• Converted with llama.cpp convert_hf_to_gguf.py --mmproj (commit 477ac13) from the 297 visual.* tensors of the original checkpoint (24 layers, hidden 1024, patch 16, spatial merge 2, out 2560); projector type qwen3vl. For conversion only, config.json architectures was set to Qwen3_5ForConditionalGeneration; weights unchanged. Q8_0 is --outtype q8_0 from the converter.
• Runner: runner/run_gguf.py --mmproj <file> --data <json with "images"> starts llama-server --mmproj ... --image-min-tokens 64 --image-max-tokens 1600 (non-causal override as above), posts /embeddings with multimodal_data, reads the media marker from /props. Images are placed before the text; <|fim_suffix|> positions are counted from the end.
• Tests (2 synthetic text-sign PNGs, 512x256, 4 questions; 270 tokens/request), vs the original model in PyTorch fp32 with the same images:

• Text path unchanged: the new runner without images (with or without mmproj loaded) reproduced the previous Q8_0 results 246/246 with max abs dlogit 0.
• Caveat: llama.cpp decodes the image chunk as a separate batch, so non-causal attention does not span the image/text boundary as in the original model; this likely accounts for part of the logit gap. Tiny test set; not a quality benchmark.