JetBrains/Qwen3.8-3.6-27B-blend (a 50/50
linear merge of Qwen3.6-27B and Qwen3.8-27B) converted to the production LQ4 format of
ninfer-amd-r9700 for a single AMD Radeon AI PRO R9700
(gfx1201, 32 GB), with a serving setup tuned for the highest parallel throughput on real long-context
agentic work: many coding/reverse-engineering agents at once, each with up to a 112k-token context
full of tool output. The artifact carries the blend-matched
JetBrains/Qwen3.8-3.6-27B-blend-DFlash2
drafter.
Why the blend and not base Qwen3.8-27B: on our internal agent benchmark the blend solves about as many tasks as base Qwen3.8-27B but finishes each in about half the time, because it reasons much less. For agents working against a time limit, that matters most.
./build-r9700/apps/ninfer-serve ./blend-lq4/blend-r9700-lq4.ninfer \
--host 0.0.0.0 --port 8080 --model-id qwen3.8-27b-blend \
--max-context 114688 --max-concurrency 12 --kv-capacity auto --pending-timeout-ms 600000 \
--kv-cache tq-k4-v3 --spec mtp --draft-tokens 4 --lm-head-draft \
--no-p-less-sampling --temperature 1.0 --top-k 20 --top-p 0.95 \
--default-max-tokens 32768 --default-thinking-token-budget 6144 \
--kv-ram-capacity 16384 --kv-ram-system-prompts 2
tq-k4-v3 stores 15.1 KiB per token, against 19.1 for the NVFP4 default and 27.6 for
FP8/INT4, and solved as many tasks on our benchmark (below). --kv-capacity auto takes all VRAM left
after weights and runtime: about 1.04M tokens at 12 agents.--kv-ram-capacity 16384 keeps suspended and idle sessions in host RAM, so they are
restored instead of prefilled again. Its pinned memory comes from the amdgpu GTT budget, which by
default is too small: add ttm.pages_limit=7340032 (28 GiB) to the kernel command line, or leave
out the two --kv-ram flags.--lm-head-draft. Without it, ninfer drafts MTP tokens through the full 248k-token head. The
131k-token draft head is lossless, because the target model verifies every token. In our tests it
accepted the same tokens per round and was 4% faster at 12 agents and 10% faster for one session.--pending-timeout-ms 600000 keeps them queued instead of failing at the default 30 s.--no-p-less-sampling and the flags above use the model's own generation config.
The default is p-less sampling at temperature 1.5.reasoning_effort gets the chat template's default,
which is xhigh for this blend. Agents should send low: on our benchmark it completed as
many tasks as medium and xhigh with about half the reasoning tokens.
--default-thinking-token-budget caps reasoning when the client sends no budget.Our benchmark: opencode agents work through 48 reverse-engineering tasks, 90 minutes per task,
reasoning effort low, on one R9700 at 210 W. The build was experimental 8f0a04f plus a pending
MTP change (deferred GDN fold, +3% decode).
| Agents x context | KV cache | Solved | Time for all 48 |
|---|---|---|---|
| 12 x 64k | NVFP4 | 37/48 | 2h42 |
| 12 x 64k | TQ-K4-V3 | 37/48 | 2h39 |
| 12 x 112k | TQ-K4-V3 | 37/48 | 2h26 |
| 14 x 64k * | FP8/INT4 | 36/48 | 2h33 |
| 16 x 80k | TQ-K4-V3 | 35/48 | 2h48 |
* An earlier build and agent harness.
12, 14 and 16 agents finish the 48 tasks in about the same time. 14 and 16 agents generate faster from the start (16 x 80k: 403 against 371 tok/s summed over the first 51 minutes for 12 x 64k), but each agent runs slower, so the last tasks end later. With 12 agents, more context per agent meant fewer context compactions (75 at 64k, 17 at 112k) and the shortest run. Earlier runs with identical settings scored 36 to 39, so differences of one or two tasks are noise.
The server provides /v1/chat/completions (with tools), /v1/responses and Anthropic /v1/messages.
One R9700 at a 210 W power limit, MTP with 4 drafts and --lm-head-draft. Each agent's context is
real agent tool output.
Prefill: reading a prompt. ninfer processes one prompt at a time with the whole GPU, so prefill speed does not depend on how many agents are running; the other agents pause meanwhile.
| Prompt length | Prefill speed | Time for the whole prompt |
|---|---|---|
| 8k tokens | 2,602 tok/s | 3.1 s |
| 32k tokens | 2,279 tok/s | 14.1 s |
| 64k tokens | 1,968 tok/s | 32.6 s |
Decode: generating the answer. All agents generate 1,024 tokens at once, each with 8k, 32k or 64k
tokens already in its context. Each cell is the generation speed summed over all agents, with each
agent's own speed in brackets. Context per agent is the largest each agent can have while all of them
hold a full context at once (--kv-capacity = agents x context, at least 0.5 GiB of VRAM left free),
for each of the two compact KV caches. The last context column overcommits the TQ-K4-V3 pool
(--kv-capacity auto, all free VRAM) about 1.33x, like the recommended setup; it relies on suspension
and the RAM tier when every agent fills its context at once.
| Agents | Context per agent, nvfp4 |
Context per agent, tq-k4-v3 |
tq-k4-v3, 1.33x overcommit |
Decode at 8k context | at 32k | at 64k |
|---|---|---|---|---|---|---|
| 1 | 262144 | 262144 | 262144 | 104 tok/s | 96 | 80 |
| 2 | 262144 | 262144 | 262144 | 196 (98) | 186 (93) | 162 (81) |
| 4 | 221184 | 262144 | 262144 | 341 (85) | 310 (78) | 257 (64) |
| 8 | 104448 | 133120 | 184320 | 559 (70) | 418 (52) | 333 (42) |
| 12 | 65536 | 83968 | 114688 | 705 (59) | 522 (44) | |
| 16 | 47104 | 59392 | 81920 | 749 (47) | 571 (36) |
Decode was measured on experimental 8f0a04f with the NVFP4 KV cache. With one agent, speed follows how many
MTP drafts are accepted, which varies with the sampled text: six 64k runs gave 82 to 91 tok/s. Above 12
agents total decode grows little while each agent slows down.
ninfer holds a tool call until it is complete, so a long script in a tool call can leave the stream
silent for minutes. opencode's default chunkTimeout (300,000 ms) then retries the request in a loop:
"provider": {
"ninfer": {
"npm": "@ai-sdk/openai-compatible",
"options": { "baseURL": "http://HOST:8080/v1", "chunkTimeout": 3600000, "headerTimeout": 3600000 },
"models": {
"default": {
"limit": { "context": 114688, "output": 16384 },
"options": { "reasoningEffort": "low" }
}
}
}
}
--spec dflash uses the bundled JetBrains drafter and, like MTP, is lossless. In a short
single-session test it was slower than MTP with 4 drafts and --lm-head-draft (65 against about
105 tok/s, 2.5 to 3 accepted tokens per round), so MTP is the recommendation at every agent count.
Tested with ninfer-amd-r9700 experimental at 8f0a04f on ROCm 10.1 (HIP 7.16), Ubuntu, kernel 7.0.
git clone https://github.com/jmander11/ninfer-amd-r9700 && cd ninfer-amd-r9700
git checkout 8f0a04f
export ROCM_PATH=/opt/rocm/core-10.1 HIP_PATH=/opt/rocm/core-10.1 PATH=/opt/rocm/core-10.1/bin:$PATH
cmake -S . -B build-r9700 -G Ninja -DCMAKE_BUILD_TYPE=Release -DNINFER_BUILD_APPS=ON \
-DCMAKE_PREFIX_PATH=/opt/rocm/core-10.1 -DCMAKE_HIP_COMPILER=/opt/rocm/core-10.1/lib/llvm/bin/clang
cmake --build build-r9700 --parallel 8 --target ninfer-serve
hf download aptp/Qwen3.8-3.6-27B-blend-ninfer-R9700-LQ4 --local-dir ./blend-lq4
The apps need libcurl development headers (libcurl4-openssl-dev on Ubuntu).
| File | Size | SHA-256 |
|---|---|---|
blend-r9700-lq4.ninfer |
16,752,796,160 B | 6f7ec1a58a25b408111a31138aaceb5a1e91ed5d8acfb07451e5f8b5d9a00cb2 |
blend-r9700-lq4.ninfer.conversion.json |
conversion receipt: recipe, per-object origin and hash |
The artifact loads only in ninfer-amd-r9700 on gfx1201. It is not GGUF or safetensors.
Next-token scoring of 32,768-token sequences against the BF16 blend, with the second half scored (16,383 positions per corpus):
| Corpus | Quant (KV cache) | PPL | NLL increase vs BF16 |
|---|---|---|---|
| wikitext-2 | BF16 | 5.372 | |
| LQ4 (FP8 keys, INT4 values) | 5.461 | +0.0164 +- 0.0018 | |
| llama.cpp Q4_K_M (q8_0) | 5.421 | +0.0090 +- 0.0014 | |
| agent sessions | BF16 | 1.395 | |
| LQ4 (FP8 keys, INT4 values) | 1.401 | +0.0037 +- 0.0009 | |
| llama.cpp Q4_K_M (q8_0) | 1.401 | +0.0040 +- 0.0010 |
On agent text LQ4 matches Q4_K_M. On generic text it loses more, because it was calibrated on agent sessions and code. Both are within ninfer's accuracy tier. These runs used the FP8/INT4 KV cache; the maintainer measures the NVFP4 default at +0.1 to +0.3% perplexity over it.
ninfer's maintainer pipeline for the r9700-lq4 recipe (tools/convert/qwen3_8_27b_r9700:
build_draft_ranking, convert_q4, convert_fp8_hybrid, convert_selective_protected,
convert_fp8_capped, convert_dflash2_q4, compose_fp8_capped_dflash, convert_lq4), run on the
blend's BF16 checkpoint. It takes about 1 h 50 min on one R9700. The drafter objects come from
convert_dflash2_q4 with the JetBrains drafter and were spliced into the artifact; every other object
is unchanged, as the receipt records.
Calibration used 128 windows of 2,048 tokens: agent sessions (reverse-engineering tasks, tool calls, reasoning), decompiled pseudo-C and disassembly, llama.cpp sources and technical documentation. The calibration file is not published. Vision weights are included, as in the recipe, but were not tested.
Nearly all of the work behind this artifact is jmander11's ninfer-amd-r9700, the AMD port of NInfer by dylanbrodiefafard: the gfx1201 kernels, the LQ4 format and its recipe, the KV cache, the server and speculation. This release ran that pipeline on the blend and tuned the serving setup for multi-agent work.
Apache-2.0, as all inputs: Qwen3.6-27B / Qwen3.8-27B (Alibaba Cloud), JetBrains/Qwen3.8-3.6-27B-blend, JetBrains/Qwen3.8-3.6-27B-blend-DFlash2, NInfer and ninfer-amd-r9700.
JetBrains/Qwen3.8-3.6-27B-blend