D3-Ultra-NPU2

vllm-sr/d3-ultra (52ba9e7a; Decision 3.0, the 27B multimodal decision model of vLLM Semantic Router, Qwen3.5-27B text tower, a newer fine-tune of the d3-turbo architecture) for AMD Ryzen AI NPUs, served by FastFlowLM 1.0.11+. It answers System One requests -- choice, yes / no and score questions about a state, optionally with images and videos, with a probability for every option; box questions (where is it in an image or a video) and track questions (a box per moment over a whole video) as well -- over the same POST /v1/systemone API as the upstream package. Up to 32,768 input tokens per question. Like upstream d3-ultra it reads the state once per request and answers every question on it ("ask once, answer many"), so many questions about one input cost far less than one request per question.

flm pull d3:ultra                     # 16.6 GB download
flm serve --decision d3:ultra         # optionally next to a chat model: flm serve qwen3.5:9b --decision d3:ultra
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
  -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
       "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'

Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most 1.6 MP), shared by every question of the request. d3:lite (0.8B), d3:nano (2B) and d3:mini (4B) are the smaller siblings; they answer a single short question in well under a second, d3:ultra needs a few seconds (see Speed). d3:turbo is the previous 27B model on the same engine.

What this package supports on the NPU

GET /v1/systemone/models lists exactly what the running build accepts.

What is new against d3:turbo (both are inside this package's decision_config.json and are applied by the server as the upstream runtime does):

• prompt_layout header-v1: for a text request, the prompt of every question starts with the block Questions in this request: (all questions of the request, numbered, with their instructions and option texts), then the state and the question. The header is part of the shared state block, so it is computed once per request; it adds tokens to every request (13 to 3,875 tokens, median 219 on the test set). Requests with images or videos, and assets / box / track requests, use the plain layout, as upstream.
• temperature_by_questions: the softmax temperature is 1.046 for a request with exactly one choice / noul / score question and 1.0 for two or more; box and track answers keep the base temperature.

Requirements

• An AMD Ryzen AI processor with an XDNA 2 NPU (this package was tested on a Strix Halo machine), Linux with the XRT stack, FastFlowLM as above.
• Memory: the package is 16.6 GB on disk and about 15.9 GiB resident once loaded. Measured peak resident memory of the server process: 17.2 GiB after the load, 18.0 GiB for prompts up to 4k tokens and 27.0 GiB at 32,768 tokens (one request on a fresh server); the FLM catalog reserves a footprint of 28 GB (plus 8.6 GB of locked scratch memory). All numbers are from a 128 GB machine; smaller machines were not tested, plan for a 32 GB or larger system and do not run it next to other large models.
• Load time: about 13 s for the engine with the files in the page cache (flm serve also checks the 16.6 GB of files against their hashes the first time it sees them or after an upgrade).

What was changed

• Weights: the Qwen3.5-27B text tower of d3-ultra quantized to FastFlowLM's Q4NX format: every projection Q4_K with GPTQ-calibrated rounding (our own sequential GPTQ over all 64 layers, 400 tensors; calibration: 300 synthetic System One questions with the header-v1 layout and the model's prefix_bidir mask, 147,008 tokens, none from the Decision Index), token embedding Q8_0, norms and the Gated DeltaNet alpha / beta parameters kept in BF16.
• Vision tower: vision_weight.q4nx is made from d3-ultra's own vision encoder (27 layers, 1152 wide, 5120 out; it differs from stock Qwen3.5-27B's in every tensor and slightly from d3-turbo's), then quantized to Q4_K. It runs on FastFlowLM's existing vision kernels.
• Attention numerics (prefix_bidir): as upstream, the state block attends bidirectionally to itself, each question block attends to the state and to itself, never the other way, and the Gated DeltaNet layers stay causal. The engine runs the state once and the question blocks on its cached keys / values and recurrent state; a question's answer does not depend on the other questions of the request (the header, which lists them, is part of the state). usage.input_tokens counts the state once plus every question block, as upstream.
• Engine: none; d3:ultra runs on the same libraries and the same four kernels (mm, attn, vision_mm, vision_attn) as d3:turbo.
• Unchanged: readout.safetensors, decision_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja, LICENSE are the upstream files byte for byte (MANIFEST.sha256); NOTICE adds the modification statement. Prompts, token ids and answer objects follow the upstream runtime, with one known difference: FastFlowLM tokenizes with the model's own tokenizer.json, while upstream's transformers tokenizer drops the combining-mark class from its pre-tokenizer, so text with combining marks (for example an emoji variation selector) can have one token more upstream (3 of 212 text test requests, one token each).

Accuracy (against the upstream BF16 model)

All comparisons use the same requests: the upstream runtime in BF16 as the reference (run on an AMD iGPU, the vision tower of the image references on the CPU or in chunked BF16 on the GPU) and this package on the NPU.

The plain Q4_K quantization of the same model without GPTQ agreed on 91.5% of the text questions (cosine 0.9955), so GPTQ is what makes this package usable. Rounding noise of 4-bit weights accumulates over 64 layers, so this 27B model sits at the level of d3:turbo (95.7%) and d3:nano (95.7%) on this measure. The text set is deliberately hard (many near-ties); on the Decision Index rows below the agreement is 99%. The weakest items of the set are Japanese-language prompts (37 of 42 questions of the 20 multilingual requests agree, with the plain Q4_K package as well): non-English input is the part of the model that quantization hurts most. A request with one question is answered at temperature 1.046, as upstream.

Accuracy, BF16 minus this package: -0.3 pt (95% CI -1.3 to +0.5 over the 100 rows). The mean absolute probability difference is 0.0060. The subset is the first 100 rows of the 4,002-row seeded sample (round-robin over 43 benchmarks; about 2 rows per benchmark, so per-benchmark numbers are not meaningful; one row of 32 questions and a 31k-token state does not fit the 80 GiB iGPU in BF16 and is left out). It is a small check, not a replacement for the full suite.

Paired accuracy, BF16 minus this package: +2.8 pt (95% CI +0.0 to +5.7; d3:turbo +2.0 pt, d3:mini +2.5 pt on the same rows). The largest gaps are in BLINK (75.0% vs 83.3%) and Mind2Web (83.3% vs 88.3%); Hateful Memes moderation is near chance for both models and flips between them (61.7% vs 66.7%, 42% agreement). The image rows are limited by the vision tower: on FastFlowLM's current vision kernels the features of the d3:turbo tower differ from the BF16 tower's by a relative L2 error of about 0.3 (measured there; d3-ultra's tower is a slight further fine-tune of it). These 600 rows are the sample used for d3:turbo, so the numbers are comparable.

Box and track questions against the BF16 answers of the same requests: box on images (29 requests, 99 boxes) median edge difference 0.015 of the image size and median IoU 0.86 with BF16's box; assets (15 requests) top-1 100%, median IoU 0.95; box at a video moment (11 requests, 22 boxes) edge 0.024 and IoU 0.77; track (13 requests, 120 moments) edge 0.017, IoU 0.83, visible decision agrees on all; plain questions about videos (45 answers) top-1 97.8%. A few boxes of the image set miss the reference object entirely (the 5th percentile of the IoU is about 0), a confident box on the wrong object.

Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM)

Measured over HTTP (flm serve --decision d3:ultra, this GPTQ package, FastFlowLM pull request build with the CI-built libraries and xclbins, nothing else running on the NPU): median of 5 requests after a warm-up request (one request for 32k); "tokens" is the usage.input_tokens of the request, the header included.

The floor is about 3 s per request (the 15.5 GB of weights are streamed from memory once per pass, so a short question is memory-bound; d3:mini needs 0.5 s for the same question), then roughly 5 to 10 ms per token. The header makes the prompts a little longer than d3:turbo's for the same request (a request lists its questions once), so the same text costs somewhat more than the d3:turbo numbers. State once gives bit-identical answers to the one-pass-per-question path (checked on 55 requests / 389 answers). Over HTTP the 736 text questions of the accuracy set (212 requests) took 3.3 s per request at the median.

Requests on the same machine, one pass over the accuracy request sets (GPTQ package, medians):

Soak: 100 mixed requests (text, image and long-prompt requests, several questions per request, empty and malformed bodies) first one after the other, then the same schedule from 4 concurrent clients next to a streaming chat model: 0 status and 0 answer mismatches between the two runs (91 answered, 9 refused with 422 as expected), no crash; resident memory of the server 18.1 GB after the load, a peak of 26.1 GB, 19.6 GB afterwards.

Comparison

• Against d3:turbo (same engine): upstream reports (its own private tests) SR-Text 87.2 for d3-ultra against 83.1 for d3-turbo and SR-Box 58.2% against 60.0%. The NPU port costs the same kind of quantization noise as d3:turbo: 95.1% top-1 agreement on the hard text set (turbo 95.7%) and 99% on Decision Index rows; the vision gap to BF16 is +2.8 pt (turbo +2.0). Speed and memory are those of d3:turbo; the header adds tokens to a request.
• Against the smaller NPU siblings: d3:mini answers a 330-token question in 0.5 s and d3:nano in 0.25 s; d3:ultra takes about 3.4 s, but reads a state once for any number of questions (the NPU siblings in FastFlowLM read the prompt again for every question).
• Against the BF16 model on the same requests: see Accuracy. The quantization costs a few points of top-1 agreement on hard items, more than for d3:mini, and the image path costs somewhat more than text.

License

Apache-2.0, as the upstream model; see LICENSE and NOTICE. Built on Qwen/Qwen3.5-27B (Apache-2.0). Quantized and repackaged for FastFlowLM by the FastFlowLM team; the readout, decision config, tokenizer and chat template are unmodified.

Citation

Please cite the upstream model (vllm-sr/d3-ultra):

@misc{d3ultra_2026,
  title        = {{d3-ultra}: A Multimodal Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3-ultra}}
}