Qwen3.8-Flash-Next-Abliterated — NVFP4 (W4A16)

v2 (2026-11-03): this checkpoint now loads and serves on vLLM 0.32.0 (v1 does not). The NVFP4 weight shards are byte-for-byte those of v1; what changed is the n-gram table and the config (below). Verified text-only on one H200 with vLLM's Marlin NVFP4-MoE kernel — see Running it. v1 remains available at revision 480e00b.

NVFP4 build of windowsxp811203/Qwen3.8-Flash-Next-Abliterated, a refusal-removed Qwen3.8-Flash-Next. 335.3 GiB → 125.9 GiB.

This is weight-only NVFP4 (quant_algo: W4A16_NVFP4): activations stay in bf16, so there are no activation scales and no FP4 tensor-core compute is required. On Hopper the NVFP4 experts run as W4A16. Every v2 number on this card was measured on one H200 with vLLM's Marlin kernel; stock vLLM 0.32.0 prefers its HUMMING kernel on sm_90, which failed to JIT on our host, so that path has not been tested with this checkpoint.

What changed in v2

Files: v1's model-00008-of-00009.safetensors (the bf16 table) and config.json.text-only are removed, so the -of-00009 shards skip 00008; that is expected — model.safetensors.index.json lists all 17 weight files.

Why the n-gram swap is safe. Abliteration never touched the n-gram table: its 131 tensors (128 table shards plus three small int64 tensors) are byte-identical between the base model and the abliterated parent (31 of the 33 shard files that hold them have the same LFS sha256 on the Hub; the 7 n-gram tensors in the other two were compared one by one), so v1's bf16 table is the base model's table, and Qwen's own FP8 release of that same table is its official low-precision form. vLLM 0.32 loads it through its dedicated FP8 PLE path (Qwen4ExpPLEFp8EmbeddingMethod, logged at start-up), and every check below ran through that path. v1 kept the table in bf16 because no runtime then existed to test an FP8 path.

Community variant — credit where due. gorbatjovy got here first: qwen3.8-flash-next-abliterated-NVFP4-plefp8 found both reasons v1 would not load in vLLM and fixed them the same way — the n-gram table re-laid-out into the sharded form vLLM's loader expects and cast to FP8 with a per-tensor scale bit-identical to Qwen's own, and layer_types renamed — and served it on 2× DGX Spark. Against a BF16-table build of v1, on the first-token distribution of 80 harmless prompts: top-1 agreement 100 %, mean KL 0.0018, median 0.0006, p90 0.0060. v2 copies the FP8 table from Qwen's release instead of casting it.

Verified here (v2)

Served with vLLM 0.32.0 on one H200 (sm_90; text-only; NVFP4 experts on the Marlin kernel, see Running it); greedy, non-thinking; the same prompts, chat format and first-400-character keyword rule as the parent card, on the first 100 AdvBench and GSM8K items. GSM8K answers were generated to 1024 tokens as in the parent, refusal answers to 256. Refusal markers: the in-file list of the parent's eval_full_flashnext.py plus Traditional-Chinese and curly-apostrophe forms, whereas the parent's own numbers were scored with a different 49-entry list that script imports when available — so the two refusal columns are close but not strictly comparable:

¹ The keyword heuristic flags 2 of the 30 answers in the scored run because they use 無法 ("cannot") descriptively, e.g. "網頁 JavaScript 無法直接讀取…" inside a full, compliant answer; the flagged answers were read by hand and none refuses.

The MTP head works. On a 155-prompt mix (MT-Bench en/zh, WildChat zh, HumanEval, GSM8K and MATH-500 with and without thinking, 8K-token agent prompts with tools), num_speculative_tokens=3 reaches a mean acceptance length of 3.01 tokens per verification step (2.01 of the 3 drafted tokens accepted, plus the bonus token) and cuts median time per output token from 7.84 ms to 3.35 ms (2.34×) at one request (output throughput 124 → 258 tok/s); 598 → 1051 tok/s at 8 concurrent requests (greedy, one H200). The MTP weights ship in this checkpoint; enabling it raises the weights on the GPU from 73.6 GiB to 78.7 GiB.

What is quantized

222,747 tensors; nothing is missing relative to the bf16 parent (nvfp4_breakdown.py). The vision tower and the MTP head are excluded from quantization by the 294-entry ignore list in the shipped config.json; see v1's card (revision above) for why grafting them back and keeping them out of quantization mattered.

Running it

VLLM_USE_V2_MODEL_RUNNER=1 VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve windowsxp811203/Qwen3.8-Flash-Next-Abliterated-NVFP4 \
  --language-model-only --max-model-len 32768 --reasoning-parser qwen3

Every v2 number on this card was measured with this setup (vLLM 0.32.0, one H200 per server; each run also set its own --max-model-len and --max-num-seqs and passed --enable-auto-tool-choice --tool-call-parser qwen3_coder) plus one local edit: on sm_90, vLLM 0.32.0 prefers its HUMMING NVFP4-MoE kernel, whose JIT failed on our host (NVRTC picked up the system CUDA 12.6), so we emptied _HUMMING_PREFERRED_CAPABILITIES in vllm/model_executor/layers/quantization/utils/humming/priority.py to use Marlin (--moe-backend marlin is not a substitute with MTP on: it also applies to the bf16 MTP-head experts, which vLLM rejects for Marlin; without MTP it was not tried). VLLM_USE_FLASHINFER_SAMPLER=0 avoided a FlashInfer sampler JIT failure on the same host, so that sampler never ran; the default model runner was not tried. --language-model-only skips the vision tower; image and video input were not exercised.

• Memory. By default vLLM 0.32 keeps the 47.7 GiB FP8 n-gram table in pinned host RAM (logged as weight_device=cpu, pinned=True) and loads 73.6 GiB of weights onto the GPU (78.7 GiB with MTP). Plan for about 50 GiB of free host RAM per server. Smaller GPUs and tensor parallelism were not tested. Data parallelism on one host did not start in our two attempts (DP=3 fails vLLM's MoE intermediate_size % tp_size check; at DP=4 two of the four workers died during start-up, cause not logged), so we ran one independent server per GPU.
• Long context. Also served at --max-model-len 262144 with agent prompts of 32K–128K tokens (with MTP k=3).
• MTP. Add --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'.
• Thinking is on by default; the quality checks above turned it off per request ("chat_template_kwargs": {"enable_thinking": false}).
• Only vLLM was tested.

How it was built

v1: quantize_nvfp4_flashnext.py → graft_nvfp4.py → fix_nvfp4_config.py → verify_nvfp4.py (all in this repo).

v2: assemble_nvfp4_v2.py OURS_BODY_DIR QWEN_FP8_DIR BASE_BF16_DIR OUT_DIR (in this repo); no weight is re-quantized. OURS_BODY_DIR is this repo at 480e00b — its weight shards are hard-linked unchanged except model-00008, which is dropped. QWEN_FP8_DIR holds Qwen/Qwen3.8-Flash-Next-FP8 at 236dfdf2: its model.safetensors.index.json plus the 33 shards that contain n-gram tensors. BASE_BF16_DIR is Qwen/Qwen3.8-Flash-Next; only its config.json is read (for layer_types). The script writes the 128 FP8 n-gram shards and the scale into the eight model-ple-* files, restores layer_types, sets ple_embedding_dtype, and rewrites the index.

Parent model quality (bf16)

Full method, the λ sweep, the paired McNemar test on MMLU and the byte-level verification are in the parent model card. The architecture write-up is ARCHITECTURE.md.

License

Qwen Community License 1.0 — not Apache-2.0 — inherited from the base model. Publishing and redistributing derivative weights is permitted. Running a Model-as-a-Service or AI Work Assistant business requires a separate license from Qwen before any commercial use; internal use is exempt provided the model, its outputs and its capabilities reach no third party.

Support / 打賞

If these models are useful to you, tips are appreciated — they pay for the GPU time. 如果這些模型對你有幫助，歡迎打賞，用於支應算力成本。

USDT (TRC20) · TPTo32r7vKazpTNaFqfFZ2ztoK1DG88888

Disclaimer

This model will not refuse. It is published for alignment and safety research. You are responsible for your use of it and for complying with applicable law.