An NVFP4 quantization of perplexity-ai/pplx-decider-flash-v1.2. Selected full-attention and linear-attention gate modules retain their original BF16 weights.
| Component | Configuration |
|---|---|
| Quantization | NVIDIA ModelOpt NVFP4, Local-Hessian |
| Calibration requests | 160 |
| Retained in BF16 | 32 full-attention Q/K/V/O modules |
| Retained in BF16 | 48 linear-attention in_proj_a / in_proj_b modules |
| Decision readout | Original BF16 |
| Weight file size | 4.38 GB |
The retained gate modules are linear-attention in_proj_a and in_proj_b; MLP gate_proj is not included.
Measured on DGX Spark GB10 over 5 independent server starts per model. The machine was rebooted when switching models, and the same cache was restored for each run. Throughput is reported as mean ± sample standard deviation.
| Model | Accuracy | Throughput, concurrency 1 | Throughput, concurrency 8 |
|---|---|---|---|
| NVFP4 / Attention·Gate BF16 | 636/838 · 75.895% | 24.742 ± 0.096 req/s | 38.221 ± 0.316 req/s |
| FP8 | 648/838 · 77.327% | 20.384 ± 0.167 req/s | 30.125 ± 0.141 req/s |
| BF16 | 647/838 · 77.208% | 13.340 ± 0.077 req/s | 20.225 ± 0.302 req/s |
At concurrency 8, throughput was 26.9% higher than FP8, with accuracy 1.432 percentage points lower. The FP8 baseline was quantized by vLLM at load time from the same BF16 source.
Accuracy was measured with the custom DecisionCore evaluation, not the official Decision Index. The 2 shared over-length items were excluded from the full set of 840 questions. On the 695 questions remaining after excluding known calibration-data overlap, accuracy was 539/695 · 77.554%.
Throughput was measured after warm-up using 1,024 requests per concurrency setting. Input length averaged 370.1 tokens, with a range of 95–3,576 tokens. Per-run results are included in provenance/benchmark-summary.json.
Across 5 starts with the same cache, all selected answers and probabilities were identical for the 838 questions.
In a separate fresh-cache check, the number of correct answers was unchanged, but 2 selected answers differed. A control run using the original tuning files reproduced all original selected answers.
Validated environment: DGX Spark GB10, Linux aarch64, CUDA 13.0, vLLM 0.32.0, PyTorch 2.13.0+cu130, Transformers 5.19.0.
Run the included vLLM adapter in a compatible environment. runtime/requirements-validated.txt records the versions observed in the validated image; it is not a lock file validated by a fresh installation. Prepare CUDA builds of vLLM and PyTorch appropriate for your GPU and platform.
source/pyproject.toml preserves the upstream reference implementation and lists Torch and Transformers versions that differ from the runtime above. The included server imports source/src directly. Do not overwrite the serving environment's dependencies with pip install ./source; use runtime/requirements-validated.txt to check runtime versions.
The container image used for validation is private. A fresh dependency installation in a public image, or execution from a newly downloaded Hub snapshot, has not been validated.
bash runtime/start.sh
Once the engine is ready, run the example:
python3 runtime/example.py --base-url http://127.0.0.1:8140
The API exposes POST /v1/decisions and POST /v1/systemone. Both accept state and questions and return per-option probabilities and decisions. The current adapter supports text input.
Apache-2.0. The package includes the upstream LICENSE and NOTICE, quantization settings, and records of the modules retained in BF16.
model.safetensors 4.4 GBreadout.safetensors 1 MB