FastFlowLM build: Nemotron-3.5-Bolt-E4B-DFlash-NPU2
The DFlash speculative-decoding draft of NVIDIA's Nemotron-3.5-Bolt-E4B, converted for the AMD Ryzen AI NPU and run by
FastFlowLM (flm). It does nothing on its own: it is the second half of the
nemotron3.5-bolt-dflash:e4b tag, whose main weights are the existing nemotron3.5-bolt:e4b package
(Nemotron-3.5-Bolt-E4B-NPU2, unchanged). flm pull nemotron3.5-bolt-dflash:e4b downloads both.
flm run nemotron3.5-bolt-dflash:e4b
Needs FastFlowLM 1.0.12 or later (the release that adds DFlash and two-package tags). FLM_DFLASH=0 turns speculation off
(the tag then decodes exactly like nemotron3.5-bolt:e4b).
| Upstream | nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash (BF16 GGUF, the source of the weights here) |
| Requires | nemotron3.5-bolt:e4b (main model, tokenizer, chat template, vision and audio towers) |
| License | OpenMDW-1.1 (LICENSE.OpenMDW-1.1, kept with this package as the license requires); the Gemma 4 base is Apache-2.0 |
| Draft | 5 Qwen3-style layers, hidden 2560, FFN 5120, 32 query / 8 key-value heads of 128, attention window 1024, block of 8 tokens, 0.36 B parameters; shares the target's embedding table and output head |
| Weights | dflash_weight.q4nx: the 5 layers' matrices in Q4_1 (5 bits per weight with scales and minimums); the fc feature projection and the norms in BF16. About 0.27 GB |
| Taps | features of target layers 5, 11, 17, 23 and 35 (zero-based layer outputs), fused by the draft's fc layer |
How it is used
Each step the draft proposes up to 7 tokens in one pass; the main model checks all of them together and keeps the longest prefix it agrees with, plus one token of its own. With a sampler (temperature above 0) every position is sampled and compared with the draft, so the output distribution is the main model's.
Measured draft quality (acceptance length, tokens emitted per step)
Offline replay of this draft's reference implementation on real target features (BF16 target, thinking on, 16 MT-Bench and SPEED-Bench prompts, 400 generated tokens each, greedy):
| Draft weights | Mean acceptance length |
|---|---|
| BF16 (NVIDIA's file) | 3.83 |
| all matrices Q4_1 | 3.78 (-1.3%) |
this package (Q4_1, fc in BF16) |
3.83 |
NVIDIA reports 3.6 to 3.9 on MT-Bench and SPEED-Bench; llama.cpp with this draft gives 3.71 on our 40-prompt set, FastFlowLM 3.71.
FastFlowLM measurements
Ryzen AI MAX+ PRO 395 (128 GB), Linux, default power mode, flm serve, 512 generated tokens, 3 prompts per context, decode tokens/s.
nemotron3.5-bolt:e4b is the same main model without the draft.
| Prompt context | nemotron3.5-bolt:e4b |
DFlash greedy, think off | greedy, think on | sampled (default), think off | sampled, think on |
|---|---|---|---|---|---|
| ~30 | 13.3 | 1.74x | 1.89x | 1.59x | 1.71x |
| 1k | 12.7 | 1.54x | 1.78x | 1.51x | 1.76x |
| 4k | 11.8 | 1.33x | 1.55x | 1.27x | 1.38x |
| 16k | 9.1 | 1.43x | 1.64x | 1.40x | 1.61x |
The speedup depends on how predictable the text is (code and structured answers gain the most). The NPU checks the 8 positions of a step in
one pass; that pass computes in a different order than single-token decoding, so greedy output can differ from nemotron3.5-bolt:e4b at
near-ties of the two best tokens (97% of verified positions agree exactly with token-by-token decoding, 99.5% outside near-ties).