A lightly requantized derivative of Intel/Qwen3.8-Flash-Next-W4A16-AutoRound. The routed experts keep Intel's original 4-bit AutoRound weights (W4A16); a selected set of non-expert projections (the QSA attention and GDN linear-attention matrices, 156 in total) is converted from BF16 to symmetric per-channel 8-bit (Attn8), and the PLE lookup table is replaced with an FP8 version (FP8PLE). Since 13 Sep 2026 the checkpoint also carries a 4-bit/8-bit MTP draft head (1.33 GiB, from klee100) in place of the original BF16 MTP head (4.86 GiB), so MTP speculative decoding can be enabled in vLLM.
The Attn8 layout was chosen deliberately for the AllSpark W8A16 kernel (Alibaba's DashInfer GEMM) on Ampere-class GPUs such as the RTX 3090. vLLM removed AllSpark in #58001 (22 Sep 2026); newer builds (recipe v1.4.0 and later) serve the same INT8 weights with Marlin W8A16. Both read half the weight bytes of BF16 at decode time.
Benefits on our four-RTX-3090 setup:
Use the four-RTX-3090 vLLM recipe for setup instructions, the Docker image build, and the required patches. This checkpoint was tested with that patched build, not stock vLLM. Use the recipe's explicit Sharp chat-template setting.
MTP is off by default. To enable it, add --speculative-config={"method":"mtp","num_speculative_tokens":2} (the recipe ships
this as recipe/yml-tp2-pp-2-mtp.yml). It needs recipe v1.1.0-eed1f3d0 or later: older vLLM builds cannot run MTP under
pipeline parallelism and do not know this model's quantized draft head.
| Component | Precision | Rationale |
|---|---|---|
| Routed experts: 512 experts/layer | INT4 g128, symmetric, W4A16 | Same as Intel AutoRound; original expert weights are unchanged. |
QSA attention: self_attn.{q_proj,k_proj,v_proj,o_proj}, 12 layers |
INT8 per-channel, symmetric, W8A16 | 48 matrices converted from BF16 using RTN. Reduces weight storage and bandwidth without reducing these projections to INT4. group_size=-1, with one scale per output channel. |
GDN / linear-attention large projections: linear_attn.{in_proj_qkv,in_proj_z,out_proj}, 36 layers |
INT8 per-channel, symmetric, W8A16 | 108 matrices converted from BF16 using RTN. Only these large projections are quantized, not the entire recurrent module. |
Other GDN components: in_proj_a, in_proj_b, conv1d, A_log, dt_bias, and normalization |
BF16 | Keeps recurrent-control parameters and small components unchanged; only the large projections listed above are converted to INT8. |
| N-gram / PLE lookup table | FP8 E4M3FN + BF16 scale | Taken from albucino, originally from RadixArk. Only the lookup table is replaced; surrounding PLE processing stays BF16. |
| MTP draft head: routed experts | INT4 g128, symmetric, W4A16 (RTN) | From klee100's checkpoint (only the mtp.* shards are reused). Replaces the BF16 head shipped until 12 Sep 2026. |
MTP draft head: QSA attention, indexer, fc_embedding, fc_hidden |
INT8 g128, symmetric, W8A16 (RTN) | Same source. Router, shared expert and norms of the draft head stay BF16. |
Remaining weights: shared experts, routers, QSA indexers, token embeddings, lm_head, and vision |
BF16 | Unchanged from Intel. |
Current vLLM (recipe v1.4.0, nightly 29468dde and later): AllSpark no longer exists in vLLM
(#58001), so the 156 INT8 matrices are served by Marlin W8A16
(Using MarlinLinearKernel for AutoGPTQLinearMethod). The files on disk are unchanged. With recipe v1.4.0 the checkpoint
was verified to boot and pass its functional probes (tool calls, reasoning, prefix cache, vision); the performance tables
in this card were measured with AllSpark and have not been re-measured on Marlin.
qwen4_exp MTP module through the same INC config; its INT4 experts and INT8
projections both run on Marlin, like the body. The draft head lives on the last pipeline stage. DFlash and
DSpark are not available for this model.Historical (recipe up to v1.3.0, nightly eed1f3d0 and earlier), AllSpark:
The INT8 layout above was chosen to land on a specific kernel. On Ampere GPUs, vLLM routes the 156 INT8 matrices to
AllSpark, the W8A16 GEMM from Alibaba's DashInfer engine that was contributed
upstream and lived in the vLLM tree until #58001 (csrc/quantization/gptq_allspark/, wrapper AllSparkLinearKernel). The boot log
confirms it for every worker:
INFO [auto_gptq.py] Using AllSparkLinearKernel for AutoGPTQLinearMethod
group_size = -1, no zero points) with
per-GPU matrix dimensions divisible by 16, matching the Attn8 layout. Grouped INT8 (e.g. g128) uses a different kernel
path, such as Marlin, subject to its own shape requirements. Small GDN control projections and shared experts remain BF16.| Model | VRAM used under load per GPU | PLE lookup table in host RAM | Loaded weights per rank (1 rank/GPU) |
|---|---|---|---|
| Intel W4A16 AutoRound | ~23.4 GiB on the tightest GPU | BF16 — ~95.37 GiB | 17.93–17.98 GiB |
| W4A16-Attn8-FP8PLE | 22.64–23.00 GiB | FP8 — ~47.68 GiB | 17.82–17.87 GiB |
These are reference measurements, not a controlled memory comparison. The profiles differ as described below; the loaded-weight difference should not be read as an isolated measure of the INT8 savings. The PLE column covers only the lookup table, not total server RAM.
| Model / profile | vLLM base | KV pool (tokens, whole engine) | Cold prefill @10K (tok/s) | Cold prefill @90–100K (tok/s) | Decode, C=1 prose / code (tok/s) |
|---|---|---|---|---|---|
| Intel W4A16 AutoRound | 7c5dc571 (2 Sep) |
947,222 | 4,383 | 4,508 | ~53 |
| W4A16-Attn8-FP8PLE | e962733e (5 Sep) |
941,463 | 4,355 | 4,183 | 72.3 |
| W4A16-Attn8-FP8PLE, no speculation | eed1f3d0 (12 Sep) |
995,019–1,017,569 | 4,380 | 4,583 | 83.8 / 84.1 |
| W4A16-Attn8-FP8PLE, MTP n=2 | eed1f3d0 (12 Sep) |
526,909 | 4,404 | 4,601 | 131.8 / 160.5 |
Benchmark conditions: these are reference runs, not a controlled A/B comparison. The Intel run (2 September 2026)
used an older nightly, a text-only profile and GPU memory utilization of 0.95. The later runs used vision enabled and
utilization of 0.94; prefill depth was 90K for the e962733e row and 100K for the eed1f3d0 rows. All used TP2 × PP2.
MTP draft acceptance in the n=2 run: 70% during the benchmark (position 1: 84%, position 2: 68%; mean accepted length
2.4 of 3 tokens per step). The no-speculation KV pool is given as a range from two boots of the same configuration.
The KV pool is shared across the whole engine and varies by about ±2% between starts. Decode was measured with one prose request at a time; differences below about 11% are within this rig's observed noise. Prefill measures processing without prefix-cache reuse. All reference speeds used a P2P-patched driver; benchmark again on other setups.
quality-test.sh, 75 scenarios, pass@1)Three fresh server starts per model, alternating between Intel AutoRound (base) and W4A16-Attn8-FP8PLE, using the same
engine and configuration. Each start runs the streaming tool-call probe (60 tests, --thinking on --tool-choice both --repeat 3)
and the 75-scenario quality suite once before the server is stopped. The operating system's file cache is not cleared.
Quality tests used short prompts, one request at a time, temperature=0, max_tokens=4000, and GPU memory utilization
of 0.88 for both models, rather than the recipe's 0.94. The suite uses
club-3090's quality-test.sh --medium, scored by benchlocal-cli.
These settings differ from the standard benchmark defaults, so compare scores only with runs using the same protocol.
| Arm | Mean of 3 boots (/75) | Spread across boots | Streaming tool-call probe |
|---|---|---|---|
| Intel W4A16 AutoRound | 67.7 | 66–69 | 60/60 on all 3 boots |
| W4A16-Attn8-FP8PLE | 67.7 | 66–69 | 60/60 on all 3 boots |
Both models have the same mean (67.7/75) and range (66–69), and both passed all streaming tool-call tests on every start. No test pack differs by more than one scenario on average.
No quality difference was detected in this suite. This does not establish identical quality on every task or at long context lengths. A single run is not enough to rank models with small score differences.
Raw per-boot scores (pass@1, scenarios passed out of 15 per pack; streaming probe cells passed out of 60):
| Boot | Arm | Total /75 | Tool calling | Instruction following | Structured output | Data extraction | Reasoning / math | Streaming probe | Boot to healthy |
|---|---|---|---|---|---|---|---|---|---|
| run 1 | Intel W4A16 AutoRound | 68 | 14 | 15 | 15 | 13 | 11 | 60/60 | 452 s |
| run 1 | W4A16-Attn8-FP8PLE | 69 | 13 | 15 | 15 | 13 | 13 | 60/60 | 393 s |
| run 2 | Intel W4A16 AutoRound | 66 | 13 | 14 | 14 | 13 | 12 | 60/60 | 332 s |
| run 2 | W4A16-Attn8-FP8PLE | 68 | 12 | 15 | 15 | 13 | 13 | 60/60 | 256 s |
| run 3 | Intel W4A16 AutoRound | 69 | 13 | 15 | 15 | 13 | 13 | 60/60 | 332 s |
| run 3 | W4A16-Attn8-FP8PLE | 66 | 14 | 14 | 14 | 13 | 11 | 60/60 | 257 s |
MTP profile, 13 Sep 2026 (vLLM eed1f3d0, recipe v1.1.0-eed1f3d0, one boot, same protocol): 68/75 pass@1
(69 pass@3), streaming tool-call probe 60/60. The same two scenarios that fail without MTP (TC-11, DE-10) fail here too.
Speculative decoding with greedy verification does not change which tokens are produced; this run confirms the quantized
draft head does not break tool calling or structured output.
Still untested: hours-long stability, sustained traffic with multiple simultaneous requests, every request type at the full 262K context limit, and performance without P2P or on GPUs other than the RTX 3090.
| Contribution | Source |
|---|---|
| Original model | Qwen/Qwen3.8-Flash-Next |
| Base checkpoint and INT4 AutoRound expert quantization | Intel/Qwen3.8-Flash-Next-W4A16-AutoRound |
| FP8 E4M3FN PLE lookup table and scale | RadixArk/Qwen3.8-Flash-Next-NVFP4 (only the PLE table is used, not the NVFP4 backbone) |
| Intermediate source of the FP8 PLE files | albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE |
4-bit/8-bit MTP draft head (mtp.* shards only) |
klee100/Qwen3.8-Flash-Next-AutoRound-3bpw-MTP (AutoRound export; MTP experts W4A16 g128 RTN, MTP QSA/dense W8A16 g128 RTN) |
| Serving engine, PLE offload / FP8 KV / PP patches | vLLM and the upstream PR authors credited in the recipe |
| AllSpark W8A16 kernel that served the INT8 projections through recipe v1.3.0 | Alibaba DashInfer team, via its upstream contribution to vLLM |
| Marlin W4A16 / W8A16 kernels serving the INT4 experts and, from recipe v1.4.0, the INT8 projections | IST-DASLab and Neural Magic, via vLLM |
| Chat template | Sharp v22.4.0 by peculiar-ragdoll, building on froggeric |
| Selective INT8 RTN conversion, W4A16-Attn8-FP8PLE assembly, MTP head splice and quantization rules, four-3090 validation | todiadiyatmo |
v1.4.0-29468dde the INT8 projections run on
Marlin W8A16. Weights unchanged. Verified to boot and pass functional probes on v1.4.0; the performance numbers in this
card were measured with AllSpark and have not been re-measured.model_extra_tensors.safetensors (BF16 MTP head, 4.86 GiB) replaced by
mtp-model-00001-of-00002.safetensors and mtp-model-00002-of-00002.safetensors (1.33 GiB, from klee100).
config.json gained quantization rules for mtp.* modules (the former rule that forced the MTP head to BF16 was removed);
model.safetensors.index.json, quantization_config.json and MANIFEST.json updated. Body weights unchanged.
Tested with recipe v1.1.0-eed1f3d0 (vLLM nightly eed1f3d0), TP2 × PP2, MTP n=2: decode 132 / 160 tok/s (prose /
code), KV pool 527K, quality 68/75, tool calls 60/60. The previous revision remains
available with --revision 12c006bb0496db523dce34ca9105ee5d84a01a31.v1.0.0-e962733e (vLLM nightly e962733e).model-00005-of-00017.safetensors 5.2 GBmodel-plefp8-00001.safetensors 5.2 GBmodel-plefp8-00000.safetensors 5.2 GBmodel-plefp8-00002.safetensors 5.2 GBmodel-plefp8-00004.safetensors 5.2 GBmodel-plefp8-00005.safetensors 5.2 GBmodel-plefp8-00006.safetensors 5.2 GBmodel-plefp8-00007.safetensors 5.2 GBmodel-plefp8-00008.safetensors 5.2 GBmodel-plefp8-00003.safetensors 5.2 GBmodel-00008-of-00017.safetensors 5.2 GBmodel-00014-of-00017.safetensors 5.2 GBmodel-00011-of-00017.safetensors 5.2 GBmodel-00007-of-00017.safetensors 5.1 GBmodel-00010-of-00017.safetensors 5.1 GBmodel-00013-of-00017.safetensors 5.1 GBmodel-00004-of-00017.safetensors 5.1 GBmodel-00003-of-00017.safetensors 5.1 GBmodel-00012-of-00017.safetensors 5.1 GBmodel-00009-of-00017.safetensors 5.1 GBmodel-00006-of-00017.safetensors 5.1 GBmodel-00015-of-00017.safetensors 5.0 GBmodel-plefp8-00009.safetensors 4.4 GBmodel-00001-of-00017.safetensors 2.8 GBmodel-00017-of-00017.safetensors 1.3 GB