pplx-decider-v1-27b EXL3 4.00 bpw

EXL3 4.00 bpw conversion of perplexity-ai/pplx-decider-v1-27b.

• Source revision: 5117a6c7fe73b19308dc1a6b0fb529a40c2ecad4
• ExLlamaV3: 1.5.2+cu128.torch2.10.0
• Target bitrate: 4.00 bpw
• Architecture: Qwen3_5Model
• Safetensors size: 15.35 GB / 14.29 GiB

Important: this is a decision model, not a normal chat model

pplx-decider-v1-27b does not use the normal LM-head token-generation path for its final answer. It produces a hidden state and applies the model's custom readout.safetensors decision head to obtain option probabilities.

The repository therefore includes pplx_decider_exl3.py, which registers the model architecture and performs the custom decision readout.

TabbyAPI compatibility

This model is not recommended for standard TabbyAPI usage.

A local smoke test with TabbyAPI and ExLlamaV3 1.5.2 failed during startup with:

AssertionError: Unknown architecture Qwen3_5Model

More importantly, simply registering the architecture is not sufficient for normal /v1/chat/completions behavior: TabbyAPI's standard generation path expects an LM head and token generation, while this model requires the custom decision readout described above. A dedicated TabbyAPI backend/endpoint adapter would be needed for correct decision-model semantics.

Use the included EXL3 decision wrapper instead of treating this model as a conventional text-generation model.

RTX 4090 benchmark

Measured locally on an NVIDIA GeForce RTX 4090 24 GB with Torch 2.10.0+cu128, ExLlamaV3 1.5.2, batch size 1, FP16 cache, and direct EXL3 decision inference.

The model was kept resident in VRAM. Each context-length row below is based on 10 sequential measured requests after warm-up. Timing includes tokenization, synchronized GPU inference, decision readout, and transfer of the result back to CPU. It excludes model load time, HTTP/server overhead, and queueing.

Additional measurements:

• Model setup/load: 3.09 s with a warm OS file cache; this is not a cold-disk benchmark.
• Resident PyTorch allocation immediately after load: 12.57 GiB.
• First 120-token inference after load: 1.46 s; subsequent warm requests were much faster.
• At the end of the 8K run, nvidia-smi reported about 15.0 GiB total GPU memory use for GPU 0, including the pre-existing GPU baseline and allocator-reserved memory.

This is a classifier/decision workload, so conventional autoregressive "decode tok/s" is not the useful metric. Latency per decision and prefill/context scaling are more representative.

Validation

The EXL3 artifact passed direct local decision inference using its custom readout.

A small hand-authored sanity suite produced:

• Routing: 9/9
• Sentiment: 6/6
• Entailment: 9/9
• Arithmetic: 6/6
• Yes/no urgency: 8/8
• Explicit severity-level selection: 5/5
• Total: 43/43
• Reversing multiple-choice option order: 30/30, with the same semantic choice
• Repeating a request after intervening requests: 5/5 identical decisions, maximum probability delta 0.0

These are basic synthetic sanity checks, not an official accuracy benchmark. No BF16-vs-EXL3 accuracy/parity study has been performed, and the long-context timing prompts use synthetic filler rather than a long-context reasoning benchmark.

Notes

The original source model is a custom decision model fine-tuned from Qwen3.8-27B. Do not infer parameter count from Hugging Face's automatic architecture display alone; the source checkpoint contains about 48.6 GiB of BF16 weights, while this 4.00 bpw EXL3 artifact contains about 15.35 GB of safetensors.