Description
This Qwen3.8-Flash-Next-NVFP checkpoint contains the main model weights from nvidia/Qwen3.8-Flash-Next-NVFP4 and combined with MTP head from the official BF16 Qwen/Qwen3.8-Flash-Next. PLE table is unmodified FP8 from Qwen.
The model is resharded to reduce RAM pressure when loading the model.
This model will run on a system with 1x RTX Pro 6000 and 96gigs of system RAM. The main weights and MTP head will be on the GPU and the PLE n-gram table will be offloaded to system RAM.
No patches are needed to run this checkpoint as all the required fixes has been merged into VLLM.
Instructions
Updated Oct 29, 2026
First, pull the vllm docker nightly image:
docker pull vllm/vllm-openai:cu134-nightly-3966924d566340e971a6da313e790d05492aa505
Download the model:
hf download dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix \
--local-dir ~/models/dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix
To load the whole model you will need to increase your swap to at least 64gigs.
# create temp swap
sudo fallocate -l 64G /tmpswap
sudo chmod 600 /tmpswap
sudo mkswap /tmpswap
sudo swapon /tmpswap
swapon --show
Then drop caches and adjust swappiness to prevent the PLE table from being evicted from system RAM:
ssudo sh -c 'echo 1 > /proc/sys/vm/drop_caches'
sudo sysctl -w vm.swappiness=1
sudo sysctl -w vm.vfs_cache_pressure=200
sync && sudo sysctl -w vm.drop_caches=3
Now run vllm using:
MODEL="dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix"
MODELPATH="/root/models"
VLLM="vllm/vllm-openai:cu134-nightly-3966924d566340e971a6da313e790d05492aa505"
docker run --rm --runtime nvidia --gpus all \
-e TORCH_CUDA_ARCH_LIST=12.0f \
-e PYTORCH_ALLOC_CONF=expandable_segments:True \
-v $HOME/models:$MODELPATH/ \
-v $HOME/.cache/vllm:/root/.cache/vllm \
-v $HOME/.cache/torch:/root/.cache/torch \
-v $HOME/.cache/flashinfer:/root/.cache/flashinfer \
-p 40000:40000 \
--ipc=host \
--cap-add SYS_PTRACE \
--security-opt seccomp=unconfined \
"$VLLM" \
--model "$MODELPATH/$MODEL" \
--served-model-name qwen3.8-flash-next \
--port 40000 --host 0.0.0.0 \
--quantization modelopt \
--engram-config '{"cpu_offload": true}' \
--no-enable-flashinfer-autotune \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.95 \
--enable-prefix-caching \
--enable-chunked-prefill \
--async-scheduling \
--max-num-seqs 4 \
--max-num-batched-tokens 16384 \
--long-prefill-token-threshold 8192 \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--trust-remote-code \
--per-request-spec-decode-metrics summary \
--enable-auto-tool-choice \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_xml \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
--chat-template "$MODELPATH/$MODEL/chat_template_sharp.jinja"
Undo swap changes
# swap file remove when done
sudo swapoff /tmpswap
sudo rm /tmpswap
# swappiness undo
sudo sysctl -w vm.swappiness=60
sudo sysctl -w vm.vfs_cache_pressure=100
Benchmarks
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:--------------------------------------------------------|----------------:|------------------:|---------------:|---------------:|---------------:|----------------:|
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | pp2048 | 28981.70 ± 720.93 | | 137.04 ± 1.73 | 70.75 ± 1.73 | 137.16 ± 1.85 |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | tg128 | 170.87 ± 10.35 | 172.22 ± 10.43 | | | |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | pp2048 @ d4096 | 22047.78 ± 29.75 | | 344.99 ± 0.34 | 278.70 ± 0.34 | 344.99 ± 0.34 |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | tg128 @ d4096 | 168.98 ± 11.21 | 170.31 ± 11.30 | | | |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | pp2048 @ d8192 | 20856.87 ± 68.16 | | 557.29 ± 1.59 | 491.00 ± 1.59 | 557.29 ± 1.59 |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | tg128 @ d8192 | 169.83 ± 9.30 | 171.16 ± 9.37 | | | |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | pp2048 @ d16384 | 19106.33 ± 22.25 | | 1031.05 ± 1.09 | 964.76 ± 1.09 | 1031.05 ± 1.09 |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | tg128 @ d16384 | 171.44 ± 3.36 | 172.79 ± 3.38 | | | |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | pp2048 @ d32768 | 18597.69 ± 18.51 | | 1938.40 ± 1.86 | 1872.12 ± 1.86 | 1938.40 ± 1.86 |
| dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix | tg128 @ d32768 | 175.37 ± 3.16 | 176.75 ± 3.19 | | | |
llama-benchy (0.4.0)
date: 2026-10-29 17:43:24 | latency mode: generation
Tool Eval
╭─────────────────────────────────────────────── 🏆 Benchmark Complete ────────────────────────────────────────────────╮
│ │
│ Model: /root/models/dicksondickson/Qwen3.8-Flash-Next-NVFP4-reshard-mtp-fix │
│ Score: 96 / 100 │
│ Rating: ★★★★★ Excellent │
│ Benchmark: tool-eval-bench v3.0.1.dev4+gb672b53ff │
│ Engine: vLLM 0.33.0rc1.dev229+g3966924d5.d20261029 │
│ Quantization: NVFP4 │
│ Max context: 262,144 tokens │
│ │
│ ✅ 64 passed ⚠️ 5 partial ❌ 0 failed │
│ Points: 133/138 │
│ │
│ Quality: 96/100 │
│ Responsiveness: 89/100 (median turn: 0.7s) │
│ Deployability: 94/100 (α=0.7) │
│ Weakest: M Autonomous Planning (83%) │
│ │
│ Completed in 192.6s │
│ │
│ 📊 Token Usage: │
│ Total: 506,419 tokens │ Efficiency: 0.3 pts/1K tokens │
│ │
│ ── How this score is calculated ── │
│ • Each scenario: pass=2pt, partial=1pt, fail=0pt │
│ • Category %: earned / max per category │
│ • Final score: (total points / max points) × 100 │
│ • Deployability: 0.7×quality + 0.3×responsiveness │
│ • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Category Breakdown
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ Category ┃ Score ┃ Bar ┃ Earned ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ Tool Selection │ 100% │ ████████████████████ │ 6/6 │
│ Parameter Precision │ 100% │ ████████████████████ │ 6/6 │
│ Multi-Step Chains │ 100% │ ████████████████████ │ 8/8 │
│ Restraint & Refusal │ 100% │ ████████████████████ │ 6/6 │
│ Error Recovery │ 100% │ ████████████████████ │ 6/6 │
│ Localization │ 100% │ ████████████████████ │ 6/6 │
│ Structured Reasoning │ 100% │ ████████████████████ │ 6/6 │
│ Instruction Following │ 100% │ ████████████████████ │ 10/10 │
│ Context & State │ 100% │ ████████████████████ │ 20/20 │
│ Code Patterns │ 100% │ ████████████████████ │ 6/6 │
│ Safety & Boundaries │ 92% │ ██████████████████░░ │ 24/26 │
│ Toolset Scale │ 100% │ ████████████████████ │ 8/8 │
│ Autonomous Planning │ 83% │ ████████████████░░░░ │ 5/6 │
│ Creative Composition │ 83% │ ████████████████░░░░ │ 5/6 │
│ Structured Output │ 92% │ ██████████████████░░ │ 11/12 │
└─────────────────────────────────────────┴─────────────────┴─────────────────────────────────────────┴────────────────┘