Qwen3.6-35B-A3B NVFP4, NInfer v3 artifact

A NInfer v3 artifact of Qwen3.6-35B-A3B whose routed and shared experts are the NVFP4 weights of RedHatAI/Qwen3.6-35B-A3B-NVFP4, copied code for code, for NInfer-all, the master branch of iamwavecut/ninfer-all. NVFP4 expert banks prefill through W4A4 on Blackwell's FP4 tensor cores, so this file needs an sm_120a GPU (RTX 50 series, RTX PRO 6000 Blackwell) and a 120a build of master from commit a00d1636 on (earlier commits run the banks only with -DNINFER_SM120_NATIVE=ON); sm_80, sm_86 and sm_89 builds refuse them. The file also carries incoai's DFlash2 drafter (November 10, 2026).

What is inside

component	representation
routed experts (256 per layer) and shared expert, 40 layers	NVFP4 from RedHatAI's checkpoint: FP4 codes with an FP8 scale per 16 weights and one FP32 divisor per stacked matrix, copied without requantization
attention (10 layers) and GDN (30 layers) projections	Q8, groups of 32, from the BF16 base
router, shared-expert score, GDN A/B controls, norms, convolution	BF16; `A_log` and `dt_bias` FP32
token table, output head	Q8 (groups of 32), Q6 (groups of 64)
MTP head	Q8, its experts included, from the base
DFlash2 drafter	incoai/Qwen3.6-35B-A3B-DFlash2: six sliding-window layers of width 2,048 that draft from the model's hidden states; projections Q8 (groups of 32), the convolution kernel projections, the selector and the norms BF16
proposal head	the output head's rows for the 131,072 most frequent tokens of NInfer's token ranking, Q4
Vision tower	groupwise Q4/Q5, Q8 merger, Q6 patch embedding
chat template	NInfer's pinned `qwen3_6.jinja`

One .ninfer file of 22,586,296,580 bytes (21.04 GiB; 19.13 GiB of text weights are resident without Vision and drafters), next to its conversion report, SHA256SUMS and NOTICE.

Conversion command, from the repository's tree, with the BF16 base at revision 995ad96e, RedHatAI's checkpoint at e94a930c and incoai's drafter at 51ef7b69:

python3 -m tools.convert \
  --model Qwen3.6-35B-A3B \
  --recipe qwen3_6_35b_a3b_nvfp4 \
  --source quantized=Qwen3.6-35B-A3B-NVFP4 \
  --source dflash2=Qwen3.6-35B-A3B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen3_6.jinja \
  --proposal \
  --name qwen3.6-35b-a3b \
  --out Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer

--device cpu converts without a GPU, in about four minutes.

Running

The whole 262,144-token window with MTP and Vision, from the Docker image, whose 120a build starts on these GPUs (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit):

hf download WaveCut/Qwen3.6-35B-A3B-NVFP4-NInfer-v3 Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer --local-dir models
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer \
  --model-id qwen3.6-35b-a3b --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 \
  --gdn-state-fp16 --spec mtp --draft-tokens 3 --lm-head-draft --vision

The server answers OpenAI and Anthropic requests on http://localhost:8080/v1. A build of master with CMAKE_CUDA_ARCHITECTURES=120a and CUDA 13.1 or newer (its README) takes the same flags:

ninfer-serve models/Qwen3.6-35B-A3B-NVFP4-ninfer-v3.ninfer --model-id qwen3.6-35b-a3b \
  --max-context 262144 --kv-capacity 262144 --kv-dtype rk8v4 --gdn-state-fp16 \
  --spec mtp --draft-tokens 3 --lm-head-draft --vision

On an RTX 5090 this starts in 17 seconds and leaves 7.4 GiB of the card free.

November 10, 2026: DFlash2 drafter

This update adds the DFlash 2 draft model of incoai/Qwen3.6-35B-A3B-DFlash2 (Apache-2.0, revision 51ef7b69) as the file's dflash2 component: 108 bindings, 690,772,992 bytes. The file was converted again with the command above; every one of the previous file's 32,632 parameters is bound to byte-identical objects, and the text, Vision and MTP configurations are equal. A round proposes its drafts in one pass of the drafter and the model verifies them, so every committed token is the model's own.

--spec dflash2 --draft-tokens N selects the drafter in a build of master from commit ebaad2e26 on. Release 2026-11-09.2 and earlier builds load the file, run it without drafts and with MTP as before, and refuse --spec dflash2. The drafter was trained on blocks of eight positions: use at most seven drafts.

On an RTX 5090 (575 W, CUDA 13.4, BF16 KV, everything resident; 400 greedy tokens, two runs each, which agreed within 1%) the answer to a code prompt, the answer to a prose prompt and 128 tokens of the benchmark corpus (ninfer_bench, three repetitions) decoded at:

	code prompt	prose prompt	benchmark corpus
no drafts	403 tok/s	403 tok/s	403 tok/s
MTP, 3 drafts	829 tok/s	548 tok/s	497 tok/s
DFlash2, 3 drafts	805 tok/s	465 tok/s	427 tok/s
DFlash2, 5 drafts	919 tok/s	420 tok/s	350 tok/s
DFlash2, 7 drafts	892 tok/s	385 tok/s	455 tok/s

DFlash2 is ahead of MTP only on the code prompt (11% at five drafts) and behind it on prose at every count; seven drafts are slower there than no drafts. MTP with three drafts stays the recommended command. On an RTX 5060 Ti with the experts in host memory (256 tokens) the code prompt decoded at 103.5 tok/s without drafts, 163.4 with MTP and 176.9 with seven DFlash2 drafts, and the prose prompt at 117.3, 132.6 and 114.4.

Quality

Perplexity over the quick subset of NInfer's evaluation corpus (ninfer-perplexity --quick, 261,167 scored tokens, INT8 KV), both artifacts on the same RTX PRO 6000:

model	perplexity
this artifact	4.410
official Qwen3.6-35B-A3B NInfer artifact (groupwise Q4/Q5 experts)	4.364

An RTX 5090 scores this artifact at the same 4.410.

Speed

September 2026, ninfer_bench on NInfer's benchmark corpus, BF16 KV, greedy, three repetitions, the default 120a build with each card's built-in device profile:

	RTX 5090 (550 W)	RTX PRO 6000 (600 W)	official artifact, RTX PRO 6000
prompt, 512 tokens	21,622 tok/s	21,705 tok/s	18,040 tok/s
prompt, 4,096 tokens	27,412 tok/s	30,938 tok/s	22,643 tok/s
decode, 128 tokens	407 tok/s	399 tok/s	398 tok/s
decode with MTP (3 drafts, proposal head)	541 tok/s

The MTP row drafts over the benchmark corpus's prose (42% of drafts accepted); a short chat answer decodes at 742 tok/s on the RTX 5090. Decode reads about the same bytes as the official artifact, so it runs at the same speed; prompts gain from the FP4 tensor cores.

License

Apache 2.0, as every source; see NOTICE.