D3-Edge-NPU2
vllm-sr/d3-edge (Decision 3.0, the decision model of vLLM Semantic
Router; Qwen3.5-0.8B architecture cut to 12 layers, revision c454ee39) for AMD Ryzen AI NPUs, served by
FastFlowLM 1.0.12 or later. It answers System One requests -- choice, yes / no
and score questions about a state, optionally with images, with a probability for every option -- over the same
POST /v1/systemone API as the upstream package. Up to 32,768 input tokens per question. Video is not supported (a
request with videos is refused with HTTP 422).
flm serve --decision d3:edge # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:edge
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
-d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
"criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'
Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most
1.6 MP), shared by every question of the request. d3:lite (0.8B), d3:nano (2B), d3:mini (4B) and d3:flash (9B)
are the larger siblings.
What was changed
- Weights: the 12-layer text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with
GPTQ-calibrated rounding (calibration: 192 synthetic System One prompts, none from the Decision Index), token
embedding Q8_0, norms and the Gated DeltaNet parameters unquantized.
vision_weight.q4nxholds the vision tower (Qwen3.5-0.8B's, merger included), used for requests with images. - Unchanged:
readout.safetensors,decision_config.json,tokenizer.json,tokenizer_config.json,chat_template.jinjaandLICENSEare the upstream files byte for byte (MANIFEST.sha256). - Same contract: prompts, token ids and answer objects follow the upstream runtime.
Accuracy (against the upstream BF16 model)
| this package on the NPU | upstream BF16 | |
|---|---|---|
| Decision Index public-suite sample, mean benchmark score (1,600 requests, 37 benchmarks) | 49.8% | 47.2% |
| Top-1 agreement with BF16 on the same requests (4,452 choice answers) | 93.8% | -- |
| Long prompts, 835 to 31,545 tokens (8 requests), same top answer as BF16 | 6 of 8 | -- |
| Decision Index public vision suite subset, accuracy (300 questions, 10 benchmarks) | 53.0% | 55.7% |
| Top-1 agreement with BF16 on the same image questions | 82.3% | -- |
Mean benchmark score, this package minus BF16: +2.6 pt (standard error 1.1 pt; about 42 requests per benchmark, so single benchmarks are noisy), i.e. no accuracy loss is measurable. Answers agree with BF16 less often than for the larger sizes (93.8%; plain Q4_K without GPTQ: 91.9%): a small model has many near-tied options, and the two long prompts that differ are such near-ties. The sample is a seeded round-robin over the public suite's requests whose prompts are at most 8,000 tokens (37 of its 40 benchmarks have such requests); RouterBench and SGD requests are left out because their locally rebuilt payloads differ from the lab's. The BF16 scores are the upstream BF16 run's own answers on the same requests. The vision subset is 30 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); paired accuracy, BF16 minus this package: +2.7 pt (95% CI -0.7 to +6.0 pt), so images may cost a little accuracy.
Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.12)
| request | median latency |
|---|---|
| 1 question, 128 tokens | 60 ms |
| 1 question, 330 tokens | 83 ms |
| 1 question, 1,024 tokens | 0.26 s |
| 1 question, 4,096 tokens | 1.0 s |
| 1 question, 16,384 tokens | 6.0 s |
| 1 question, 31,545 tokens | 16 s |
| 1 question, 32,767 tokens | 17 s |
| 4 questions, 330 tokens each | 0.29 s |
| 16 questions, 330 tokens each | 1.0 s |
| 1 question + a 1024 x 768 image (880 tokens, HTTP) | 0.61 s |
| 1 question + two images (HTTP) | 0.89 s |
| 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 1.4 s |
| 8 questions + one image (one image encoding, HTTP) | 2.0 s |
From 1,536 tokens attention runs on the NPU's attention kernel. Prompts over 32,768 tokens are answered with
max_length_exceeded (nothing is truncated). Resident memory of the server: 2.0 GB after loading, up to 4.8 GB at
32,768 tokens with images in the mix. Median of 3 (16,384 tokens and longer: 1 to 2 runs) over HTTP on an otherwise idle
machine.
License
Apache-2.0, as the upstream model; see LICENSE and NOTICE. Built on Qwen/Qwen3.5-0.8B.