Qwen3.8-Flash-Next IQ3E-Q8D-MTP (GGUF) For V100(32GB)x2

This is a single-file GGUF of Qwen/Qwen3.8-Flash-Next with the model's multi-token-prediction (MTP) head added back in. The mix of quantizations is chosen so that every routed expert fits in 2× 32 GB of VRAM, and the MTP head lets llama.cpp decode speculatively. This model and patch is specifically tuned for dual Nvidia Tesla V100(32GB VRAM).

This file needs a patched llama.cpp. It uses the qwen4exp architecture with MTP (nextn) tensors. Stock llama.cpp, Ollama and LM Studio cannot load it. See Runtime.

Where the tensors come from

• unsloth GGUF revision c8b5954a88c2775c546b92593eda40ea041d3176.
• Qwen checkpoint revision de4b8e4d43b917e7706784d8bb445c9af86a3540. Only the mtp.* tensors are taken from it.

unsloth's GGUFs have no MTP head. The head was converted separately with convert_hf_to_gguf.py --mtp, which streams only the mtp.* tensors (about 8 GB) from the original checkpoint. It was quantized to Q8_0 and then appended to the trunk. The routed experts and the per-layer embedding (PLE) table come from UD-IQ3_XXS. The other non-expert tensors come from UD-IQ4_XS, whose dense tensors are Q8_0. On V100 these run faster in small-batch GEMVs than Q6_K does.

Runtime

[!WARNING] The file loads only with a patched llama.cpp v0.5.0. Stock llama.cpp cannot load it.

The patches are in patches/:

[!CAUTION] v4 and v5 patches were published briefly and then withdrawn because they had regressions. Use v3. The withdrawn patches are still in the repo history (commit 21dbbdb8).

All versions add:

• qwen4exp support, including the FreeToken MoE port. The FreeToken sources are Apache-2.0 and their license is in patches/LICENSE-freetoken.
• MTP speculative decoding for this model.
• The V100 (sm_70) performance work: sparse QSA attention, fused kernels, and GPU argmax.

v2 adds:

• Tensor split (--split-mode tensor) for this model. Both GPUs work on every layer, and their partial results are summed by an in-graph AllReduce that writes straight into the peer GPU over NVLink or PCIe peer-to-peer. Decode is about 20% faster than with layer split.
• Fixes that let tensor split handle long prompts. v1 aborted on prompts over about 1,000 tokens in this mode.
• Faster fused hyper-connection kernels for up to 8 tokens, and MTP draft passes padded to one fixed shape.
• An even expert split under tensor split: 320 + 320 of the 640 expert columns, where 128-column steps gave 256 + 384.

v3 adds:

• Image input at the full 256K context. The vision encoder does not fit next to the 256K context, so it is loaded on demand: for each image it borrows 2 GB of model weights on GPU 0, which are released while the encoder runs and copied back afterwards. See Image input.
• MTP with images. With v2, the server aborted about 200 tokens into any answer about an image.
• Fast decode with an image in the context. The sparse attention's fast path now works on contexts that hold an image. Without it, an image in the context halved decode speed at 56K and cut it to a third at 133K.
• A fix for a race in a fused kernel. In some layouts the kernel wrote its output over an input it was still reading. At long context with images this gave NaN scores and could crash the server. It also cost tensor split its perplexity gap against layer split (see Quality notes).

Text decode speed and greedy text output are the same as with v2.

Build

git clone --branch v0.5.0 https://github.com/ggml-org/llama.cpp
cd llama.cpp
hf download pentacoxian-dev/Qwen3.8-Flash-Next-IQ3E-Q8D-MTP-GGUF \
  patches/llama-v0.5.0-v100-v3.patch patches/LICENSE-freetoken --local-dir .
git apply patches/llama-v0.5.0-v100-v3.patch

# NCCL: GGML_CUDA_NCCL is on by default, but CMake must find it. One way is the pip wheel:
pip install nvidia-nccl-cu12==2.27.3
NCCL_HOME=$(python -c 'import nvidia.nccl; print(list(nvidia.nccl.__path__)[0])')

# CUDA 12.8 or later is required (the FreeToken code uses cuda_fp4.h).
# Volta needs CUDA 12.x: CUDA 13 cannot build for sm_70.
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=70 -DGGML_NATIVE=ON -DLLAMA_BUILD_TESTS=OFF \
  -DNCCL_INCLUDE_DIR=$NCCL_HOME/include -DNCCL_LIBRARY=$NCCL_HOME/lib/libnccl.so.2
cmake --build build --target llama-server

Set CMAKE_CUDA_ARCHITECTURES to your GPU's architecture. With a pip-installed NCCL, add $NCCL_HOME/lib to LD_LIBRARY_PATH before starting the server.

The decode AllReduce is the patch's own kernel and does not use NCCL, but it needs peer access between the two GPUs. The benchmarked build was linked with NCCL. If NCCL is missing, CMake prints a warning and llama.cpp falls back to its internal AllReduce for the other reductions. That setup was not benchmarked.

To upgrade a tree that has the v2 patch applied, download patches/llama-v0.5.0-v100-v2-to-v3.patch instead, git apply it on top, and rebuild. From v1, apply patches/llama-v0.5.0-clean-20260924-to-v2.patch first.

Download

hf download pentacoxian-dev/Qwen3.8-Flash-Next-IQ3E-Q8D-MTP-GGUF \
  Qwen3.8-Flash-Next-IQ3E-Q8D-MTP.gguf --local-dir models/merged

Serve

These are the launch settings used for the numbers under Performance, on 2× V100 32 GB:

export GGML_CUDA_P2P=1 GGML_CUDA_GRAPH_OPT=1 GGML_CUDA_DISABLE_GRAPHS=1 \
       GGML_FT_GRAPH_CACHE=1 GGML_FT_GRAPH_CACHE_MIN_NODES=32 \
       LLAMA_QWEN4EXP_HC_FUSION=1 LLAMA_HC_FUSED=2 LLAMA_MTP_PAD=5 \
       LLAMA_MTP_N_UBATCH=32 LLAMA_GPU_ARGMAX=1

build/bin/llama-server \
  --model models/merged/Qwen3.8-Flash-Next-IQ3E-Q8D-MTP.gguf \
  --alias Qwen3.8-Flash-Next \
  --host 127.0.0.1 --port 8080 --api-key <your-key> \
  --ctx-size 262144 --parallel 1 \
  --threads 48 --threads-batch 48 \
  --batch-size 2048 --ubatch-size 256 \
  --n-gpu-layers auto --fit on --fit-target 512 \
  --split-mode tensor --main-gpu 0 --numa distribute \
  --moe-expert-cache 0 --moe-expert-cache-inserts 2 \
  --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --load-mode mmap \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja --reasoning off --reasoning-budget 0 --no-reasoning-preserve \
  --metrics

Compared with v1, this adds LLAMA_HC_FUSED=2, LLAMA_MTP_PAD=5 and GGML_CUDA_DISABLE_GRAPHS=1, and changes --split-mode layer --ubatch-size 512 to --split-mode tensor --ubatch-size 256. At startup the server logs two warnings that are expected under tensor split: llama_params_fit is not implemented for SPLIT_MODE_TENSOR and backend sampling not supported with SPLIT_MODE_TENSOR.

To use layer split instead (for example, if the GPUs have no peer access), keep the same environment and use --split-mode layer --ubatch-size 512.

Image input

The vision projector is unsloth's mmproj-F16.gguf (0.9 GB) at the same revision as the tensors above:

hf download unsloth/Qwen3.8-Flash-Next-GGUF mmproj-F16.gguf \
  --revision c8b5954a88c2775c546b92593eda40ea041d3176 --local-dir models/vision

export MTMD_ON_DEMAND=1 GGML_CUDA_EVICT_MIB=2048   # in addition to the environment above
# the same llama-server command as above, plus:
  --mmproj models/vision/mmproj-F16.gguf

The 256K setup leaves no room on the GPUs for the encoder, so the two switches load it on demand:

• the encoder's weights stay in host RAM;
• for each image, the last 2 GB of the model weights on GPU 0 are released and the encoder runs in that memory;
• afterwards the encoder frees everything and the weights are copied back to the same addresses. They were copied to host RAM once (2 GB of pinned memory).

Each image costs 0.2–0.5 s extra. Text requests are not affected. The encoder runs on the first GPU (-mmdev is not needed).

Without the switches, --mmproj needs a smaller context (about 128K) so the encoder can stay on the GPU. --no-mmproj-offload keeps the 256K context but runs the encoder on the CPU: 3 s for a small image, 5.5 minutes for a 4,000-token screenshot.

Command-line flags

Environment switches

Memory. Every routed expert is on the GPUs, and only the 27 GB PLE table stays in host RAM. At 262,144-token context with micro-batch 256, each GPU uses 32.3 GB.

Performance

These results come from runs on 2026-09-26 with the launch settings above: 2× Tesla V100-SXM2-32GB (an NVLink pair in a DGX-1), 2× Xeon Platinum 8168, CUDA 12.9.1 (sm_70), llama.cpp v0.5.0 with patches/llama-v0.5.0-v100-v2.patch.

Settings: 256K context, one slot, greedy decoding, non-thinking mode, 384 generated tokens, MTP with 3 draft tokens.

A speculative step is one 4-token verify plus 3 draft passes. Tokens per second depend on how many drafts the text accepts: code takes about 3.3 tokens per step and prose about 2.5. The time per step does not depend on the text, so it compares better across runs. Over three server launches the short-prompt step time was 24.0, 24.2 and 26.1 ms. The 250K row comes from the 26.1 ms launch.

On a wider set of 16 prompts (320 tokens each), tensor split decodes at 126–127 tok/s (24.5–24.7 ms per step) and layer split at 107–108 tok/s (28.7–28.9 ms per step).

These numbers were measured with v2. v3 measured the same: 24.3 ms per step on the short prompts, with or without the vision encoder loaded, and 124.6–127.3 tok/s on the 16-prompt set.

With images

v3 with the image-input settings above, 256K context. The two images are a newspaper photo and an IDE screenshot (tools/mtmd/test-1.jpeg and docs/android/imported-into-android-studio.jpg in llama.cpp).

v2 could not run MTP with images. Without MTP, decode after an image ran at about 60 tok/s.

Split mode and interconnect

Same host, same v2 build, short prompts and a 64K cache:

Tensor split needs peer access between the GPUs, but not NVLink: over PCIe it lost only about 4% on short prompts. Layer split prefills faster, including short follow-up prompts (about 500 against 280–330 tok/s).

For comparison, stock UD-IQ4_XS with experts partly in host RAM and no MTP decodes at 30.2 tok/s (llama-bench tg128, on the Xeon 8260 host).

Quality notes

• These are low-bit experts (IQ3_XXS). Expect some quality loss compared with the higher unsloth quants.
• Tested: a 3-case deterministic API smoke test (marker, arithmetic, multi-turn recall) passes with MTP enabled, under both split modes. The patched CUDA kernels match the CPU backend in test-backend-ops. Prompts of 2.4K, 12.9K, 14.5K and 250K tokens work under tensor split.
• Perplexity on 4K-token chunks: tensor split 2.8506–2.8520 with v3, layer split 2.8525. v2's tensor split gave 2.8608 (+0.3%). That gap was not summation order, as this card said before. It was the fused-kernel race fixed in v3.
• Image prompts are not bit-for-bit reproducible between runs (text is). Blocks of identical image tokens tie at the sparse attention's cut-off, and which one wins can change. The answers were the same in testing.
• MTP drafts are verified by the target model. With greedy decoding, speculation does not change the output. Kernels that change the summation order can shift near-tied tokens.

License

The weights are derived from Qwen/Qwen3.8-Flash-Next and are released under the same Qwen Community License 1.0. The full text is in LICENSE, and it must be included with any copy. Commercial products or services above 100M monthly active users or US$20M monthly revenue must display the model name prominently. See the license for the exact terms.

The patches in patches/ are not covered by the model license. They modify llama.cpp, which is MIT-licensed. Their FreeToken-derived files (freetoken-*, marked SPDX-License-Identifier: Apache-2.0) are under Apache-2.0, and that license is in patches/LICENSE-freetoken.