Qwen3.8-27B-NVFP4-DGX-SPARK-V2 + 64k MTP

This repository combines:

the complete Qwen3.8-27B NVFP4 checkpoint, including its MTP tensors;
the 64k-vocabulary MTP recipe from autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK.

The 64k MTP optimization is not baked into the safetensors weights. It is a runtime patch that must be loaded by the included vLLM plugin or by the offline Python patch.

What the 64k MTP patch does

Restricts the draft head from 248,320 vocabulary rows to 64,026 kept rows.
Pads the draft head to 64,064 rows for the quantized GEMM kernel.
Gathers the draft head from the target lm_head and quantizes it at runtime to Marlin NVFP4 W4A16.
Quantizes the five MTP linear projections at runtime to Marlin NVFP4.
Leaves the target model and the verification lm_head untouched.

The result is a much smaller draft step, allowing larger speculative token counts. The verified starting point is num_speculative_tokens: 7 on B200 and 8 on DGX Spark.

Important: 64k MTP requires the included plugin. Running a plain vllm serve command with only --speculative-config uses native MTP, not the 64k/Marlin draft path.

Requirements

vLLM 0.31.0 (the plugin uses vLLM internals and may break on other versions)
Blackwell GPU: B200 (sm_100) verified; DGX Spark / GB10 (sm_120/121) is the recipe's target platform
Python 3.10+
Enough memory for the 20.6 GB checkpoint plus KV cache
PyTorch and CUDA matching the vLLM 0.31.0 build

This MTP patch is for vLLM. It is not an SGLang or TensorRT-LLM plugin.

1. Download the model

export MODEL_DIR="$HOME/models/Qwen3.8-27B-NVFP4-DGX-SPARK-V2"

HF_HUB_ENABLE_HF_TRANSFER=1 \
hf download autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2 \
  --local-dir "$MODEL_DIR"

cd "$MODEL_DIR"

2. Install vLLM and the MTP plugin

Use a clean virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip

pip install "vllm==0.31.0"
pip install -e ./mtp64k/plugin

Verify that the plugin is installed:

python -c "import spark_mtp_draft; print(spark_mtp_draft.__file__)"

3. Set the 64k MTP environment

export SPARK_DRAFT_HOME="$PWD/mtp64k/scripts"
export SPARK_DRAFT_VOCAB="$PWD/mtp64k/artifacts/draft_vocab_64k.pt"
export SPARK_DRAFT_HEAD_Q=marlin
export SPARK_DRAFT_MTP_Q=marlin

SPARK_DRAFT_VOCAB is also the activation switch. If it is empty, the plugin stays inactive. SPARK_DRAFT_HOME can be omitted when the plugin was installed with pip install -e ./mtp64k/plugin from this repository layout, but setting it explicitly is recommended.

4. Serve with vLLM

B200

vllm serve "$MODEL_DIR" \
  --served-model-name qwen3.8-27b-nvfp4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":7,"use_local_argmax_reduction":true}' \
  --gpu-memory-utilization 0.60 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

DGX Spark / GB10

Use the same command but start with num_speculative_tokens: 8:

vllm serve "$MODEL_DIR" \
  --served-model-name qwen3.8-27b-nvfp4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":8,"use_local_argmax_reduction":true}' \
  --gpu-memory-utilization 0.60 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

0.60 GPU memory utilization is the safe starting point on unified-memory Spark systems. On a discrete B200, raise it only after confirming that the model and KV cache fit. Increase --max-model-len only if the deployment has enough KV-cache memory.

Confirm that 64k MTP is active

The server log must contain a line like:

[real-draft] head rows=64064 head_q=marlin mtp_q=marlin layers=5

If that line is absent, the plugin did not activate. The model may still start with native MTP, but the 64k draft vocabulary and Marlin draft head are not in use.

5. Send an OpenAI-compatible request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.8-27b-nvfp4",
    "messages": [{"role": "user", "content": "Write a Python LRU cache."}],
    "temperature": 0,
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

Thinking mode is enabled by the chat template by default. Disable it per request with chat_template_kwargs.enable_thinking: false.

Keep client max_tokens below the served --max-model-len; the prompt and completion must fit in the same context.

Offline Python usage

For the in-process vLLM API, apply the patch before creating LLM, and disable vLLM V1 multiprocessing so the patch is present in the engine process:

import os
import sys

os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"
sys.path.insert(0, "./mtp64k/scripts")

import real_draft
real_draft.install(
    "./mtp64k/artifacts/draft_vocab_64k.pt",
    "marlin",
    "marlin",
)

from vllm import LLM, SamplingParams

llm = LLM(
    model=".",
    speculative_config={
        "method": "mtp",
        "num_speculative_tokens": 7,
        "use_local_argmax_reduction": True,
    },
    gpu_memory_utilization=0.60,
    max_model_len=32768,
    kv_cache_dtype="fp8",
)

out = llm.chat(
    [{"role": "user", "content": "Reply with exactly: OK"}],
    SamplingParams(temperature=0, max_tokens=16),
)
print(out[0].outputs[0].text)

Run this from the downloaded repository root.

Tuning notes

B200: start at num_speculative_tokens: 7.
DGX Spark / GB10: start at num_speculative_tokens: 8.
Sampling temperature above ~0.7: reduce to roughly 4–6 speculative tokens and re-benchmark; higher entropy lowers draft acceptance.
Quantization choices: marlin is the verified default. The scripts also expose cutlass, fp8, and bf16; use them only after testing on your exact vLLM build.
Context length: 32768 is the conservative starting point. Raising it increases KV-cache memory; --kv-cache-dtype fp8 is recommended.
DGX Spark OOM: keep --gpu-memory-utilization 0.60, lower --max-model-len, or reduce concurrency.

Repository layout

.
├── config.json / tokenizer* / chat_template.jinja
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model-mtp-fc.safetensors
├── model.safetensors.index.json
├── mtp64k/
│   ├── artifacts/
│   │   └── draft_vocab_64k.pt
│   ├── scripts/
│   │   ├── real_draft.py
│   │   ├── nvfp4_ops.py
│   │   └── spark_mtp_draft.py
│   ├── plugin/
│   │   ├── pyproject.toml
│   │   └── spark_mtp_draft.py
│   └── manifest.json
├── LICENSE
└── README.md

mtp64k/manifest.json contains the packaged file hashes and recommended runtime settings.

Troubleshooting

The server starts but there is no `[real-draft]` line

Check all of the following:

python -c "import spark_mtp_draft; print(spark_mtp_draft.__file__)"
test -f "$SPARK_DRAFT_VOCAB" && echo "vocab exists"

Then restart the server with the environment from section 3. Environment variables must be inherited by the vLLM engine process.

MTP runs but there is no expected speedup

Confirm head rows=64064 appears in the log.
Confirm use_local_argmax_reduction: true is present in the speculative config.
Try a lower temperature and re-benchmark.
Compare with a single stream; batching changes the relative benefit.

CUDA driver error 803

On the B200 host used for validation, the host driver needed the system CUDA driver library to be preloaded:

export LD_PRELOAD=/lib/x86_64-linux-gnu/libcuda.so.1

This is host-specific. Do not apply it blindly if your driver stack does not need it.

Validation

The packaged repository was served directly from its download directory with vLLM 0.31.0 on a B200. The MTP patch loaded successfully:

[real-draft] head rows=64064 head_q=marlin mtp_q=marlin layers=5

An OpenAI-compatible smoke request returned OK with finish_reason=stop.

License and sources

Apache-2.0. See LICENSE.

64k MTP recipe: autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK
Qwen3.8 model family: Qwen/Qwen3.8-27B

autotrust/Qwen3.8-27B-NVFP4-DGX-SPARK-V2