FlashMap-Qwen3.8-27B-FP8

Project page · Code · Paper · Models

FlashMap is a drafter for lossless speculative decoding. It proposes a whole block of K tokens in a single pass. Unlike discrete diffusion drafters, which predict each token of the block independently and ignore inter-token dependencies, FlashMap uses a flow map (Flow Map Language Models), which jumps from Gaussian noise to the whole block in a single step and can capture dependencies between its tokens.

This is the FlashMap drafter for Qwen/Qwen3.8-27B-FP8, with 770M parameters. It was trained on the target's own responses, in non-thinking mode, to the prompts of open-perfectblend, and it drafts over the target's 65,536 most frequent tokens (its draft vocabulary). Every other token gets draft probability 0, so decoding stays lossless.

Serve it with vLLM or SGLang. The transformers reference implementation in the flashmap package (spec_generate) does not support this target: Qwen3.8's linear-attention layers keep a state that rejected drafts cannot be rolled back from.

Serve with vLLM

The vLLM integration is a plugin in the flashmap package (the vllm extra), which vLLM loads automatically.

uv venv --python 3.12 .venv-vllm
source .venv-vllm/bin/activate
uv pip install "vllm==0.32.*" "flashmap[vllm]>=0.2.1"

vllm serve Qwen/Qwen3.8-27B-FP8 --max-model-len 8192 --max-num-seqs 64 \
  --speculative-config '{"model": "j-shen/FlashMap-Qwen3.8-27B-FP8", "num_speculative_tokens": 16, "draft_sample_method": "probabilistic"}'

Qwen3.8 samples at temperature 1 by default, which needs "draft_sample_method": "probabilistic". --max-num-seqs must stay at or below the number of the target's linear-attention state blocks (about 600 on one H200 at --max-model-len 8192). If vLLM fails at startup compiling DeepGEMM's FP8 matmuls, see the troubleshooting notes.

At high concurrency, vLLM keeps a copy of the target's linear-attention state for every speculative token of every request, which limits how many requests run at once: with 16 speculative tokens and flashmap 0.2.1 or later, about 11 requests of 8,192 tokens fit on one 80 GB H100 (about twice as many as with 0.2.0). Lowering "num_speculative_tokens" (for example to 8; the drafter still drafts its whole block) fits more, at some cost in tokens per verification.

Prefix caching works (--enable-prefix-caching). On this hybrid target vLLM caches in blocks of 864 tokens and recomputes the last cached block for the drafter, so a shared prefix starts to be reused only after about 1,700 tokens. SGLang's radix cache reuses prefixes at token granularity.

Serve with SGLang

The SGLang integration is a plugin in the flashmap package (the sglang extra), which SGLang loads automatically.

uv venv --python 3.12 .venv-sglang
source .venv-sglang/bin/activate
uv pip install "sglang==0.5.20" "flashmap[sglang]>=0.2"

sglang serve --model-path Qwen/Qwen3.8-27B-FP8 \
  --speculative-algorithm DSPARK --speculative-draft-model-path j-shen/FlashMap-Qwen3.8-27B-FP8

Sending requests

Both servers expose an OpenAI-compatible API, SGLang on port 30000 and vLLM on port 8000. The drafter was trained with thinking disabled, which is where it drafts best.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="none")  # vLLM: port 8000
response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[{"role": "user", "content": "How many positive whole-number divisors does 196 have?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)

Loading the weights

The model code ships with the weights, so the drafter loads without the flashmap package:

import torch
from transformers import AutoModel

draft = AutoModel.from_pretrained("j-shen/FlashMap-Qwen3.8-27B-FP8", trust_remote_code=True, dtype=torch.bfloat16)

License

The model weights are released under Apache-2.0, following Qwen3.8. The model code (*.py, from the GitHub repository) is released under the MIT license.

Citation

@misc{flashmap2026,
  title         = {{FlashMap}: Flow Map Speculative Decoding},
  author        = {Shen, Jeff and Zhou, Zhejian and Lee, Ryan and May, Jonathan and Ho, Shirley},
  year          = {2026},
  eprint        = {2610.24365},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.24365}
}