The checkpoint · Speed · Quality · What it fits on · Use it · The base model · Bench card

Qwen3.8-27B-W4A16-Merlin

A 4-bit build of Qwen3.8-27B, Qwen's dense 27B model with vision, made to run at its full 262,144-token context on one, two or four 24 GB GPUs. It is 16 GiB instead of the original 52 GiB: the attention, Gated DeltaNet and feed-forward projections are AWQ INT4, the lm_head and the MTP draft head INT4, and the vision tower and embeddings stay in BF16. It is served by the Merlin engine, a vLLM 0.32.0 fork tuned for Ampere and Ada that keeps the token embeddings in host RAM and the KV pool in VRAM. Measured on RTX 3090s; the engine also ships profiles for other Ampere and Ada cards, from RTX 4090s and 48 GB workstation cards to A100s and CMP 170HX (see What it fits on).

_{numbers: Merlin 3.0.1 on 4× RTX 3090 w/o NVLINK @ 220W, P2P on, with the optional GPU unlock, 2026-11-02 bench card (runs 2026-11-01 to 2026-11-02, on release candidates just before the final commit); the hero is the four-card layout. KV pool: FP8, all in VRAM; seven 262K sessions held at once on four cards (3.0.0 run). Agents: eight 16K conversations, then eight simultaneous 512-token follow-ups, thinking off; the four-card total comes from an overlap window under 5 s, so read it loosely. Decode: one stream, thinking on (effort medium, the default), exactly 261,888 prompt tokens + 128 out, median of 3. Prefill: one cold 101,845-token prompt. Each figure is also in a table below.}

The checkpoint

Files in this repository

_{About 16 GiB in all (the 11 files of the engine's models/qwen38-27b/checkpoint.sha256). Also in the repository: LICENSE (Apache-2.0) and NOTICE.}

Speed

_{One stream, thinking on (effort medium), ≤ 256 tokens out (128 at 262K), median of 3; 4,046 / 32,566 / 130,386 / 261,888 prompt tokens. Thinking off at 262K: 50 / 109 / 149 tok/s on one / two / four cards. On four and two cards that is lower than in 3.0.0 (158 and 111 tok/s, −5.7 % and −2.1 %), while the 259K thinking-off ladder cells read +5.4 % and +4.8 %: the time per decode step is the same or shorter (four cards 20.00 ms against 20.78 ms), and the gap is MTP draft acceptance over a 128-token sample (2.84 against 3.12 tokens per step on four cards). The thinking-on +11.0 % at 262K on four cards is mostly the same effect in the other direction: draft acceptance over the same 128-token sample (4.41 against 4.13 tokens per step). The engine wheel is the same file.}

_{Time to first token (TTFT): the median of 3 cold sends below 262K (the thinking-on ladder cells) and the cold send 1 at 262K. Four cards at 262K: 124.7 s, against 164.1 s in 3.0.0. Prefill speed at 10K, 100K and 262K is in the next table.}

_{Prefill and time to first token, one cold request per prompt, 8 tokens out; the 261,888-token row is the exact-262K prompt's cold thinking-on send 1 (128 tokens out; the 2026-11-01 run).}

_{N 16K conversations, then N simultaneous 512-token follow-ups, total tok/s while all N decode; thinking off unless stated. One card runs two sequences at most. The 8-agent rows are 3.0.1 (2026-11-01 to 2026-11-02); with thinking on a 512-token follow-up may end inside the reasoning, and reasoning tokens count, so those totals measure decoded tokens, not finished answers. Overlap windows: four cards 4.30 s thinking off and 3.55 s on, two cards 6.08 s and 4.92 s; read the totals from windows under 5 s loosely. The 2- and 4-agent rows were not re-run: 3.0.0 runs (one and two cards: a pre-release engine build, 2026-10-17; four cards: the release wheel, 2026-10-20), windows 2.6 to 5.7 s.}

_{N needle prompts (thinking off, greedy) sent at once, then re-sent one by one: every re-send served from the prefix cache means the pool held all N. 0 preemptions. 3.0.0 run, not re-run for 3.0.1: the KV pool and the sessions it holds are unchanged.}

Against the previous release (3.0.0, one, two and four cards)

_{3.0.0 → 3.0.1 (change), tok/s; decode is one stream with thinking on at effort medium on both releases unless stated, prefill is one cold send. Cold time to first token at 262K: 574.1 → 452.4 s on one card, 305.3 → 234.9 s on two, 164.1 → 124.7 s on four. Where the settings differ: the 3.0.1 runs used the optional GPU-unlock host upgrade and the 3.0.0 runs did not (in a stock-versus-unlocked A/B on two cards at 262K the unlock alone made prefill about 29 % faster, with decode unchanged and identical output); the engine wheel is the same file; the 3.0.0 two-card 8-agent total came from a pre-release engine build (2026-10-17), the four-card one from the release wheel. Both four-card agent totals come from windows under 5 s. The 3.0.0 card has the full 3.0.0 tables.}

How it was measured

• Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02; the 3.0.1 runs also with the optional Zagitta's gpu-unlock host upgrade.
• Engine: Merlin 3.0.1 (the engine wheel is unchanged from 3.0.0), served by merlin serve with the shipped defaults, one fresh boot per layout. One card: TP1; two cards: TP2 on GPUs 0 and 2; four: TP4. Runs 2026-11-01 to 2026-11-02, on release candidates just before the final commit, not on the final commit itself.
• Layout: MTP K=4, FP8 KV, RecoverSSM on every layout, host-resident token embeddings, thinking effort medium (the server default).
• Decode: T=0, a needle prompt at depth, every send a cold prefill except sends 2 and 3 at 262K, which reuse the prefix before the needle; one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
• Streams are client-timed. Full tables, thinking-off rows and ITL: the bench card.

Quality

This checkpoint, measured on the Merlin engine:

_{Thinking-on GSM8K-200: Merlin 3.0.1, four cards, 2026-11-01 to 2026-11-02, the same 200 problems and scorer as the thinking-off row, temperature 0. Every other row is a 3.0.0 run, not re-run for 3.0.1. Four cards and images: the Merlin 3.0.0 release wheel, 2026-10-20. One and two cards' GSM8K and 131K / 262K needle panel: a pre-release engine build, 2026-10-17; the exact-261,888-token needle on every layout: the release wheel, 2026-10-20. These are checks of this checkpoint, not a matched BF16 comparison: no BF16 run of the same checks was made. The FP8 KV scales were calibrated on traffic that included the first 50 GSM8K test questions (activation maxima only); the lm_head and MTP head were GPTQ-calibrated on public benchmark prompts (ARC, MMLU-Pro, GPQA, GSM8K train split, HumanEval), synthetic tasks and a small hand-written set. No calibration data is distributed.}

The base model, as reported by Qwen for the BF16 weights:

_{A selection from the upstream card, which has the full tables and comparisons with other models. These were not re-run on this 4-bit build.}

What it fits on

_{Measured on the reference host: the KV pools as in the 2026-11-02 bench card; the sessions held at once and the image figures are 3.0.0 runs (2026-10-20 bench card), not re-run for 3.0.1. KV cache FP8, the pool all in VRAM. Two and four cards need GPU peer-to-peer on RTX 3090s for these rows (hosts without it: the no-P2P profiles below). A 4096 × 2048 image (8,388,608 px) left at least 886 MiB free on two cards and 862 MiB on four. One card takes one 4 MP image per prompt: with the single-card memory settings, one and two requests at once and after a 262K needle left at least 558 MiB free (2026-10-21). The original one-card run failed at 8 MP; with those settings 8 MP still stays below the 524 MiB headroom floor (478 MiB on the reference host).}

Other GPUs. The engine also ships profiles for these; the Status column says how far each has been run:

_{merlin serve and ./setup.sh pick the profile from the GPUs they find; without P2P the default is the largest layout, and merlin serve prints the --tp 1 --instances N command, which gives more total throughput for many users. A ~ is the engine's estimate, not a measurement. Every profile uses an FP8 KV cache, on sm_80 cards too (FP8 KV ran on the 4× CMP 170HX 64 GB rental on one, two and four cards, needles exact, GSM8K-50 48/50). 🟢 measured by us · ⚫ a profile ships, not yet qualified by us (expected to work; a rented-host run is noted in the row where one was made). Every profile: the engine's hardware matrix. 🟡 Testing: full-model runs passed on cloud-hosted cards; not yet qualified against the reference host.}

Use it

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
./setup.sh --model 27b

One command does the rest: after a small uv and Python bootstrap it checks the GPUs, driver, RAM and disk, shows the engine, package and checkpoint downloads with their sizes before fetching them, installs the prebuilt engine, downloads and verifies this checkpoint, and starts an OpenAI-compatible server on 127.0.0.1:8100. Enter takes the recommended answer at each question; --yes takes them all, --dry-run shows the plan. Run ./setup.sh again to start the server, and git pull && ./setup.sh to update the engine without downloading the model again. Requests use the model name qwen38-27b; the setup asks how many cards to use and recommends the largest tested layout (on a multi-card host without P2P it recommends --tp 1 --instances N).

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

Manual install

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \
  bash release/install-env.sh ./venv

venv/bin/hf download halt95/Qwen3.8-27B-W4A16-Merlin --revision 7a0af39f63dd4fb0733c23d3ec126762074a503f \
  --local-dir ~/models/27b
(cd ~/models/27b && sha256sum -c "$OLDPWD/models/qwen38-27b/checkpoint.sha256")
venv/bin/python merlin_cli.py serve ~/models/27b

MERLIN_WHEEL installs the prebuilt engine (sm_80, sm_86, sm_89; the same wheel file as in Merlin 3.0.0), checked against release/merlin-wheel.sha256; without it the installer compiles the engine instead, about 1.5 hours on 8 cores. merlin serve recognises the checkpoint, picks the largest tested layout for the GPUs it finds (or, with none, the largest untested one that fits, labelled as such) and serves on 127.0.0.1:8100. --tp 1|2|4 picks a layout. Add --dry-run to print the resolved command without starting anything. The first start compiles kernels.

Recommended settings (Qwen's, for Qwen3.8-27B):

_{Qwen's recommendation for this model, with min_p=0.0 and repetition_penalty=1.0 in both modes. A presence_penalty between 0 and 2 reduces endless repetition; higher values can mix languages.}

• Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
• Reasoning effort: Merlin serves medium by default; pick another per request with "chat_template_kwargs": {"reasoning_effort": "low"} (low / medium / xhigh).
• KV RAM cache (experimental, off by default): --kv-ram-cache SIZE on merlin serve or ./setup.sh keeps the prefix KV that the GPU evicts in pinned host RAM and restores it, with the same output, instead of recomputing it. It adds no context or sessions; SIZE comes on top of the host RAM above (at least 9g on one card, 10g on two, 12g on four). On four cards --swarm raises the session cap from 24 to 32 and sets a 48g cache. Under heavy concurrent prefill a session may recompute instead of restoring (output identical).
• --spec dflash2 is the Flash-Next family's drafter option; it is refused on this checkpoint, which runs its own MTP head.
• Chat template: Merlin serves Froggeric's fixed Qwen3.8 template (v22.5) by default, not the file in this repository. It accepts system messages after the first turn, so coding agents' per-turn notes stay in place and the prefix cache keeps hitting. CHAT_TEMPLATE=<path> serves another template; CHAT_TEMPLATE= (empty) serves this repository's own.
• With Qwen's own template (CHAT_TEMPLATE=), earlier turns' reasoning is kept by default and "chat_template_kwargs": {"preserve_thinking": false} keeps only the latest.
• Images: up to 64 per prompt at 8 MP each on two or four cards; one card takes one image per prompt at 4 MP. The cap is a served default: a client that raises it per request goes past the measured headroom.
• Tool calls use the standard tools field, parsed with the Qwen3 coder tool parser.
• To reach it from other machines, answer network at setup's Access question (it generates an API key), or start merlin serve with HOST=0.0.0.0 and VLLM_API_KEY set; put a TLS reverse proxy in front before opening it beyond your own network.

The base model

Qwen3.8-27B is Qwen's dense model on the Qwen3.5 architecture, with vision. The architecture, in short:

• Hybrid attention. Three Gated DeltaNet (linear-attention) layers for every gated-attention layer: 48 and 16 of the 64 layers.
• Dense. All 27B parameters work on every token; no experts.
• Multi-token prediction. One MTP layer, trained over several steps; Merlin uses it for speculative decoding (4 draft tokens).
• Thinking by default, with a non-thinking mode per request and reasoning_effort levels; Qwen's template keeps earlier turns' reasoning by default (preserved thinking).

_{Architecture as reported by Qwen on the upstream card.}

Other builds

_{One stream, thinking on, 261,888 prompt tokens + 128 out, median of 3; Merlin 3.0.1, 2026-11-02 bench cards. Effort as each bench card ran it: medium on every Qwen3.8 checkpoint (Flash-Next, Swift1.5, Swift1.5-Huihui, ThinkingCap and Qwen3.8-27B), the server default. Nemotron and Ornith 1.5 ran at full thinking: neither template reads reasoning_effort.}

Good to know

• Every figure we measured for this checkpoint comes from 4× RTX 3090 at 220 W, PCIe Gen4 with P2P, used as one, two or four cards, except the rental rows under Other GPUs; the 3.0.1 figures were also measured with the optional GPU-unlock host upgrade, the 3.0.0 ones were not; the Other builds table names each build's card count.
• The same engine profiles serve ThinkingCap, BottleCap AI's fine-tune of this model, which reasons in fewer tokens (under a different licence).
• Coming from the qwen38-27b-3090s engine (v1.x)? It is replaced by the Merlin engine; this checkpoint's files are unchanged.

Changes

_{Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.}

Credits and licence

Qwen (model; Apache-2.0, see LICENSE), philbert (the AWQ INT4 body), froggeric (the fixed Qwen3.8 chat template the engine serves), the vLLM project, FlashInfer, compressed-tensors, aikitoria (peer-to-peer on consumer Ampere), Neiko2002, and CoolShades (the idle-power report behind the v2.5.2 engine fix). Upstream fixes are credited in the engine repository.