The checkpoint · Speed · Quality · What it fits on · Use it · The base model · Bench card

Qwen3.8-Flash-Next-W4A16-Merlin

A 4-bit build of Qwen3.8-Flash-Next, Qwen's 125B mixture-of-experts model (6B active per token, with vision), made to run at its full 262,144-token context on four 24 GB GPUs. It is 115 GiB instead of the original 335 GiB: the routed experts are INT4, the 51B-parameter n-gram table is FP8, and attention and the vision tower stay in BF16. It is served by the Merlin engine, a vLLM 0.32.0 fork tuned for Ampere and Ada that keeps the n-gram table in host RAM and the whole KV cache in VRAM. Measured on RTX 3090s; the engine also ships profiles for other Ampere and Ada cards, from RTX 4090s and 48 GB workstation cards to A100s and CMP 170HX (see What it fits on).

_{numbers: Merlin 3.0.1 on 4× RTX 3090 w/o NVLINK @ 220W, P2P on, with the gpu-unlock host upgrade, 2026-11-02 bench card (runs of 2026-11-01 to 2026-11-02 on release candidates just before the final commit). KV pool: FP8, all in VRAM; three 262K sessions held at once (3.0.0 run). Agents: eight 16K conversations, then eight simultaneous 512-token follow-ups, thinking off. Decode: one stream, thinking on (effort medium, the server default), exactly 261,888 prompt tokens + 128 out, median of 3. Prefill: one cold 101,875-token prompt. Each figure is also in a table below.}

The checkpoint

Files in this repository

_{115.3 GiB in total. There are no shards 00002 and 00016: model.safetensors.index.json maps no tensor to them, so nothing is missing.}

Speed

_{4× RTX 3090, thinking on at reasoning effort medium (the server default). The needle-prompt ladder (4,046 / 32,566 / 130,386 prompt tokens, every send a cold prefill) and the exact 261,888-token prompt (send 1 cold; sends 2 and 3 reuse the prefix before the needle). Decode: ≤ 256 tokens out (128 at 262K), median of 3. Time to first token: the median of 3 cold sends below 262K, the cold send 1 at 262K. The 262K needle was found in 3 of 3 thinking-off sends (3.0.0 run).}

_{One cold request per prompt, 8 tokens out at 10K and 100K. The 261,888-token row is the 2026-11-01 run's cold thinking-on send, so its time to first token (38.5 s) can differ slightly from the 262K row above (38.4 s). The 101,875-token figure is the hero's prefill number.}

_{N 16K conversations, then N simultaneous 512-token follow-ups, total tok/s while all N decode; thinking off unless stated. With thinking on a 512-token follow-up may end inside the reasoning and reasoning tokens count, so that total measures decoded tokens, not finished answers. Overlap windows: 4.55 and 5.22 s for the 3.0.0 rows, 5.90 s (thinking off) and 5.43 s (thinking on) for 3.0.1: a few seconds each, so read these rows loosely.}

_{N needle prompts (thinking off, greedy) sent at once, then re-sent one by one: every re-send served from the prefix cache means the pool held all N. Peak pool use 100 %, 0 preemptions. 3.0.0 run, 2026-10-20; the KV pool and the sessions it holds are unchanged in 3.0.1.}

Against the previous release (3.0.0)

_{tok/s. All 3.0.1 rows ran with the gpu-unlock host upgrade on; the 3.0.0 rows did not. The engine wheel is unchanged, and the engine's stock-versus-unlocked A/B on Flash-Next found about 16 % faster prefill with the unlock and decode unchanged. Thinking-on decode also compares different efforts: 3.0.0 ran at low, 3.0.1 at medium (the server default). The 3.0.1 ladder prompts are 30 tokens shorter than the 3.0.0 ones (4,046 against 4,076 at 4K); the bench card sets them side by side.}

How it was measured

• Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02, with the Zagitta's gpu-unlock host upgrade on (the 3.0.0 runs did not have it).
• Engine: Merlin 3.0.1 (the engine wheel is the same as in 3.0.0), served by merlin serve with the shipped defaults, one fresh boot per layout. The runs were on 2026-11-01 to 2026-11-02, on release candidates just before the final commit; none used the final commit.
• Layout: TP2 × PP2 + EP on four cards, MTP K=3, FP8 KV, BF16 draft lm_head, thinking effort medium, the server default (the 3.0.0 numbers were measured at low).
• Decode: T=0, a needle prompt at depth; every send a cold prefill except sends 2 and 3 at exactly 262K, which reuse the prefix before the needle; 3 sends per cell, median. Thinking-on rows were re-measured on 2026-11-02 on a fresh boot of a later release candidate; thinking-off rows, prefill and the thinking-off agent rows are from the 2026-11-01 runs, some on an earlier candidate. tok/s = tokens after the first streamed event / time from first to last event.
• Streams are client-timed. Full tables, thinking-off rows and ITL: the bench card.

Quality

This checkpoint, measured on the Merlin engine:

_{Thinking-on GSM8K-200: Merlin 3.0.1, four cards, the same 200 problems and scorer as the thinking-off row, temperature 0. Every other row is a 3.0.0 run, not re-run for 3.0.1. GSM8K and images: the Merlin 3.0.0 release wheel, 2026-10-20. Needle panel: a pre-release engine build, 2026-10-17. These are checks of this checkpoint, not a matched BF16 comparison: no BF16 run of the same checks was made, so how much quality the 4-bit build keeps is not quantified here.}

The base model, as reported by Qwen for the BF16 weights:

_{A selection from the upstream card, which has the full tables and comparisons with other models. These were not re-run on this 4-bit build.}

What it fits on

_{Measured on the reference host (3.0.0 run, 2026-10-20 bench card): three 262K and six 131K sessions held at once with every re-send served from cache; a 4096 × 2048 image (8,388,608 px) with at least 1,026 MiB free on every card.}

Other GPUs. The engine also ships profiles for these; the Status column says how far each has been run:

_{merlin serve and ./setup.sh pick the profile from the GPUs they find. A ~ is the engine's estimate, not a measurement. "(8 at once)": the KV pool holds 17 full 262K conversations, and the server admits 8 requests at a time; the rest queue. On sm_80 cards (A100, A30, CMP 170HX) this checkpoint defaults to a BF16 KV cache, so fewer sessions fit; KV_DTYPE=fp8 opts into FP8 (untested ⚫). 🟢 measured by us · ⚫ a profile ships, not yet run by us (expected to work). Every profile: the engine's hardware matrix. 🟡 Testing: full-model runs passed on cloud-hosted cards; not yet qualified against the reference host.}

Use it

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
./setup.sh --model flash-next

One command does the rest: after a small uv and Python bootstrap it checks the GPUs, driver, RAM and disk, shows the engine, package and checkpoint downloads with their sizes before fetching them, installs the prebuilt engine, downloads and verifies this checkpoint, and starts an OpenAI-compatible server on 127.0.0.1:8000; requests name the model flash-next-v2 (also flash-next). Enter takes the recommended answer at each question; --yes takes them all, --dry-run shows the plan. Run ./setup.sh again to start the server, and git pull && ./setup.sh to update the engine without downloading the model again.

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

Manual install

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \
  bash release/install-env.sh ./venv

venv/bin/hf download halt95/Qwen3.8-Flash-Next-W4A16-Merlin --revision cec53e9f90e5995b81691604ec0b32191debf797 \
  --local-dir ~/models/flash-next
(cd ~/models/flash-next && sha256sum -c "$OLDPWD/models/flash-next/checkpoint.sha256")
venv/bin/merlin serve ~/models/flash-next

MERLIN_WHEEL installs the prebuilt engine (sm_80, sm_86, sm_89; the same wheel file as in Merlin 3.0.0), checked against release/merlin-wheel.sha256; without it the installer compiles the engine instead, about 1.5 hours on 8 cores. merlin serve recognises the checkpoint, picks the largest tested layout for the GPUs it finds (or, with none, the largest untested one that fits, labelled as such) and serves on 127.0.0.1:8000. Add --dry-run to print the resolved command without starting anything. The first start compiles kernels.

Recommended settings (Qwen's, for this model):

_{Qwen's recommendation for this model, with min_p=0.0 and repetition_penalty=1.0 in both modes. A presence_penalty between 0 and 2 reduces endless repetition; higher values can mix languages.}

• Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
• Reasoning effort: Merlin serves medium by default; pick another per request with "chat_template_kwargs": {"reasoning_effort": "low"} (low / medium / xhigh).
• Chat template: Merlin serves Froggeric's fixed Qwen3.8 template (v22.5) by default, not the file in this repository. It accepts system messages after the first turn, so coding agents' per-turn notes stay in place and the prefix cache keeps hitting. CHAT_TEMPLATE=<path> serves another template; CHAT_TEMPLATE= (empty) serves this repository's own.
• With Qwen's own template (CHAT_TEMPLATE=), earlier turns' reasoning is kept by default and "chat_template_kwargs": {"preserve_thinking": false} keeps only the latest.
• Images: up to 42 per request (the launcher's cap), 8 MP each; four 8 MP images in one prompt were checked on the Flash-Next checkpoint.
• Tool calls use the standard tools field, parsed with the Qwen3 coder tool parser.
• To reach it from other machines, answer network at setup's Access question (it generates an API key), or start merlin serve with HOST=0.0.0.0 and VLLM_API_KEY set; put a TLS reverse proxy in front before opening it beyond your own network.

Optional Merlin 3.0.1 features, all off by default (flags and limits):

• KV RAM cache (experimental). --kv-ram-cache 24g keeps conversation prefixes the GPU evicts in host RAM and restores them instead of recomputing them, with exactly the same output as a GPU cache hit. On Flash-Next a 262K restore takes 1.2–1.4 s instead of about 45 s to recompute. It works with the MTP drafter only (not with --spec dflash2) and adds no context or sessions.
• More sessions with --swarm (experimental). Raises the session cap on four RTX 3090s from 8 to 24, with a 48 GB KV RAM cache. Tested on this checkpoint with P2P; without P2P it uses the same cap untested (⚫, expected to work).
• DFlash2 drafter. --spec dflash2 on four RTX 3090s with P2P decodes 5–18 % faster than the default MTP drafter across the Flash-Next family; the server then holds three 262K sessions instead of six 131K ones. ./setup.sh --spec dflash2 downloads the 1.2 GB drafter.
• INT6 n-gram table. --ple int6 loads a smaller INT6 copy of the n-gram table and saves about 9 GiB of host RAM per server, with quality checks passed on this checkpoint and on Swift1.5. It is one shared pack laid over the FP8 files you downloaded, which stay untouched; FP8 stays the default.
• Past 262K with YaRN. No profile turns YaRN on; you add it as a vLLM override, and asking for more than 262K without it is refused at startup. Tested on the reference host at 393K and 524K (MTP and DFlash2) and 917K (MTP); 1M is untested. The recipes are in How to serve.

The base model

Qwen3.8-Flash-Next is Qwen's preview of the architecture behind Qwen4. What changed, in short:

• Hybrid attention. Three Gated DeltaNet (linear-attention) layers for every Qwen Sparse Attention layer. QSA attends over micro-blocks instead of single tokens, which keeps long-context latency down.
• Sparse experts. 512 experts per layer with 10 routed and 1 shared active per token: 6B of the 125B parameters do the work for each token.
• N-gram embedding. A 51B-parameter bigram/trigram table at layer 2 adds parameter capacity with less computation than adding experts would. Merlin keeps it in host RAM, so the VRAM goes to KV cache.
• Gated residual. Widened residual streams with a data-dependent read gate and a per-branch write gate.
• Multi-token prediction. One MTP layer, trained over several steps; Merlin uses it for speculative decoding.
• Thinking by default, with a non-thinking mode per request; Qwen's template keeps earlier turns' reasoning by default (preserved thinking).

_{Architecture as reported by Qwen on the upstream card.}

Other builds

_{One stream, thinking on, 261,888 prompt tokens + 128 out, median of 3; Merlin 3.0.1, 2026-11-02 bench cards. Thinking on at reasoning effort medium, the server default, wherever the checkpoint has effort levels; the Nemotron and Ornith 1.5 templates have none, so those two ran at full thinking.}

Good to know

• Every figure we measured for this checkpoint comes from one setup: 4× RTX 3090 at 220 W, PCIe Gen4 x16 with P2P (the 3.0.1 figures also with the gpu-unlock host upgrade on), except the rented-host row under Other GPUs; upstream results and the Other builds rows use their stated setups. Hosts without P2P use the no-P2P profiles (see What it fits on).
• Only if you shrink the KV pool yourself: keep it larger than --max-model-len plus a few blocks. With the shipped settings this cannot occur.
• Coming from the qwen38-flash-next-3090s engine (v2.x or earlier)? It is replaced by the Merlin engine; this checkpoint's files are unchanged.

Changes

_{Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.}

Credits and licence

Qwen (model; Qwen Community License 1.0, see LICENSE), Intel (AutoRound experts), RadixArk (FP8 PLE table), DominikBucko (MTP packing recipe), froggeric (the fixed Qwen3.8 chat template the engine serves), the vLLM community PLE-offload work, aikitoria (peer-to-peer on consumer Ampere), Neiko2002, brandon-nuttall (YaRN past 262K report), R-omk (YaRN past 262K crash report and recipes), and a community idle-power report (behind the v2.5.2 engine fix). Upstream fixes are credited in the engine repository.