D3-Lite-NPU2

vllm-sr/d3-lite v3.1.3 (Decision 3.0, the decision model of vLLM Semantic Router) for AMD Ryzen AI NPUs, served by FastFlowLM 1.0.11+. It answers System One requests -- choice, yes / no and score questions about a state, optionally with images, with a probability for every option -- over the same POST /v1/systemone API as the upstream package. Up to 32,768 input tokens per question.

flm serve --decision d3:lite          # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:lite
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
  -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
       "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'

Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most 1.6 MP), shared by every question of the request:

curl -sLo panda.png https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png
printf '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?",
  "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}},
  "images": ["data:image/png;base64,%s"]}' "$(base64 -w0 panda.png)" > image_request.json
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" -d @image_request.json
# ... "choice":"panda", "panda":0.999 ...

What was changed

Weights: the Qwen3.5 text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with GPTQ-calibrated rounding (calibration: 300 Decision Index prompts, none of them in the evaluation sets below), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. vision_weight.q4nx holds the vision tower (BF16), used from FastFlowLM 1.0.11 for requests with images.
Unchanged: readout.safetensors, decision_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja and LICENSE are the upstream files byte for byte (MANIFEST.sha256).
Same contract: prompts, token ids and answer objects are byte-identical to the upstream runtime (checked on every FastFlowLM build against 1,365 upstream-rendered requests, 57 of them with images); image grids and input token counts equal upstream's, and the image pixels are within 2 grey levels of the upstream processor's.

Accuracy (against the upstream BF16 model)

	this package on the NPU	upstream BF16
Decision Index public-suite subset, gold accuracy (6,824 questions, 35 benchmarks)	60.5%	61.2%
Top-1 agreement with BF16 on the same questions (7,226)	90.6%	--
Decision Index public vision suite, gold accuracy (600 questions, 10 benchmarks)	61.7%	60.2%
Top-1 agreement with BF16 on the same image questions	84.5%	--

Paired gold accuracy, upstream BF16 minus this package: text +0.67 pt (95% CI [+0.08, +1.27] pt); images -1.50 pt (95% CI [-3.92, +0.92] pt). The vision suite leaves out Winoground (gated); per-benchmark numbers are on the FastFlowLM d3 benchmark page.

Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.11)

request	median latency
1 question, 128 tokens	109 ms
1 question, 330 tokens	151 ms
1 question, 1,024 tokens	430 ms
1 question, 4,096 tokens	1,587 ms
1 question, 16,384 tokens	8,425 ms
1 question, 32,768 tokens	22,696 ms
4 questions, 330 tokens each	536 ms
16 questions, 330 tokens each	2,067 ms
1 question + a 640 x 480 image (398 tokens)	393 ms
1 question + a 1.6 MP image (1,698 tokens)	1,529 ms
8 questions + one 1.6 MP image (one image encoding)	6,368 ms

From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster). Prompts over 32,768 tokens are answered with max_length_exceeded (nothing is truncated). Image rows were measured over HTTP; the others at the engine (HTTP adds about a millisecond).

License

Apache-2.0, as the upstream model; see LICENSE and NOTICE.