d3-lite for the Ryzen AI NPU

vllm-sr/d3-lite, the 0.8B decision model of vLLM Semantic Router's Decision 3.0, converted to run on the Ryzen AI NPU of AMD Strix Halo inside halogen-flash-server, beside the Flash model on the GPU and behind the same port. Questions go to the Discord. Bugs go to the server repository's issues.

These files run only inside the halogen-flash-server container, on the NPU. They will not load in transformers or any other runtime.

What is here

devices/             221 MB  the NPU program for this model, and its NOTICE.md
d3-lite.hnpw        2.27 GB  the weights, converted for the NPU
vis/                 153 MB  the NPU program and weights of the model's vision tower, and its NOTICE.md
tokenizer/            13 MB  the model's own tokenizer

Use it

Start halogen-flash-server 0.18.11 or later with HALOGEN_DOWNLOAD on and HALOGEN_NPU_MODELS=d3-lite. The server fetches these files into /models/npu/d3-lite/ and checks each one's size and checksum before it starts. A request's model may name d3-lite or the upstream name vllm-sr/d3-lite.

The files at the head of this repository need halogen-flash-server 0.18.12 or later; each server version downloads the revision it was released with, so an older server keeps working unchanged.

It answers System One questions on POST /v1/systemone with the model's own contract: a state (text, or an object or array sent as JSON; an empty state reads as "(empty)") and questions, each a choice (1 to 255 options), a noul (yes or no) or a score (an ordered list of levels). Every question is one pass over the whole prompt, and the model reads the prompt in both directions, as it was trained. The answers match the model's own runtime on the CPU: on our decision benchmark the NPU picked the same answer on 509 of 512 questions, and the three it did not were near ties. Prompts go up to 8,192 tokens.

Pictures: a request may carry one picture in images (a list of one: a data: URL or base64, as a string or as {"url": ...}), and every question in the request sees it. The server sizes it as the model's own processor does, within its pixel budget (65,536 to 1,638,400 pixels), with the top lowered to 768 x 768 or the same area at another shape (2,304 patches of 16 pixels), the most the NPU's vision tower takes. Video is not served. On 64 CV-Bench questions the NPU picked the model's own CPU answer on 63; the one it did not was a near tie.

On one Strix Halo, one question served alone takes about 75 ms at a short state, 185 ms at 700 tokens and 650 ms at 3,600 tokens (the request's wall, HTTP included). A question about a picture takes about 0.2 s at 256 x 256 and 0.47 s at 768 x 768, the vision tower included.

The host needs the NPU driver, XRT with its NPU plugin, and the GPU's fabric clock held while the NPU works beside the GPU. The setup and every request shape are in docs/NPU.md.

Source and license

Converted from vllm-sr/d3-lite at revision 03560c26f5199c3c04129a2ccb38960c913635e0, itself built on Qwen/Qwen3.5-0.8B. Apache-2.0, as the source. devices/NOTICE.md lists the third-party notices the NPU program carries.