D3-Flash-NPU2
vllm-sr/d3-flash (Decision 3.0, the decision model of vLLM Semantic
Router; Qwen3.5-9B text tower, revision 13dc1cc8) for AMD Ryzen AI NPUs, served by
FastFlowLM 1.0.12 or later. It answers System One requests -- choice, yes / no
and score questions about a state, optionally with images, with a probability for every option -- over the same
POST /v1/systemone API as the upstream package. Up to 32,768 input tokens per question. Video is not supported (a
request with videos is refused with HTTP 422).
flm serve --decision d3:flash # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:flash
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
-d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
"criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'
Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most
1.6 MP), shared by every question of the request. d3:lite (0.8B), d3:nano (2B) and d3:mini (4B) are the smaller
siblings.
What was changed
- Weights: the Qwen3.5-9B text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with
GPTQ-calibrated rounding (calibration: 192 synthetic System One prompts of 100 to 2,048 tokens, none from the Decision
Index), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized.
vision_weight.q4nxholds the vision tower (identical to Qwen3.5-9B's, merger included), used for requests with images. - Unchanged:
readout.safetensors,decision_config.json,tokenizer.json,tokenizer_config.json,chat_template.jinjaandLICENSEare the upstream files byte for byte (MANIFEST.sha256). - Same contract: prompts, token ids and answer objects follow the upstream runtime.
Accuracy (against the upstream BF16 model)
| this package on the NPU | upstream BF16 | |
|---|---|---|
| Decision Index public-suite sample, mean benchmark score (1,600 requests, 37 benchmarks) | 67.4% | 67.8% |
| Top-1 agreement with BF16 on the same requests (4,452 choice answers) | 97.0% | -- |
| Long prompts, 835 to 31,545 tokens (8 requests), same top answer as BF16 | 8 of 8 | -- |
| Decision Index public vision suite subset, accuracy (300 questions, 10 benchmarks) | 72.7% | 75.0% |
| Top-1 agreement with BF16 on the same image questions | 90.0% | -- |
Mean benchmark score, this package minus BF16: -0.36 pt (standard error 0.66 pt; about 41 requests per benchmark, so single benchmarks are noisy). The sample is a seeded round-robin over the public suite's requests whose prompts are at most 8,000 tokens (37 of its 40 benchmarks have such requests); RouterBench and SGD requests are left out because their locally rebuilt payloads differ from the lab's. The BF16 scores are the upstream BF16 run's own answers on the same requests. The vision subset is 30 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); paired accuracy, BF16 minus this package: +2.3 pt (95% CI -0.9 to +5.6 pt), so images may cost a little accuracy while the answers agree with BF16 on 90.0% of the questions.
Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.12)
| request | median latency |
|---|---|
| 1 question, 128 tokens | 0.85 s |
| 1 question, 330 tokens | 1.0 s |
| 1 question, 1,024 tokens | 2.3 s |
| 1 question, 4,096 tokens | 9.0 s |
| 1 question, 16,383 tokens | 41 s |
| 1 question, 31,545 tokens | 92 s |
| 1 question, 32,768 tokens | 96 s |
| 4 questions, 330 tokens each | 3.1 s |
| 16 questions, 330 tokens each | 11.7 s |
| 1 question + a 1024 x 768 image (880 tokens, HTTP) | 3.3 s |
| 1 question + two images (HTTP) | 5.2 s |
| 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 7.0 s |
| 8 questions + one image (one image encoding, HTTP) | 18.3 s |
From 1,536 tokens attention runs on the NPU's attention kernel. Prompts over 32,768 tokens are answered with
max_length_exceeded (nothing is truncated). Resident memory of the server: 6.8 GB after loading, up to 14.1 GB at
32,768 tokens with images in the mix. Median of 3 (16,383 tokens and longer: 1 to 2 runs) over HTTP on an otherwise idle
machine.
License
Apache-2.0, as the upstream model; see LICENSE and NOTICE. Built on Qwen/Qwen3.5-9B.