Qwen3.8-27B NVFP4 on one RTX 5090 (NInfer)
Serving recipe for Qwen3.8-27B in NVFP4 with MTP or DFlash2 speculative decoding on a single RTX 5090, using the
NInfer engine. This repository holds no weights. The weights are the
unmodified artifact neroued/Qwen3.8-27B-nvfp4-NInfer.
Scripts, pins and benchmark: nvfp4-kernels/ in goethe.
Pinned inputs
| item | value |
|---|---|
| engine | https://github.com/Neroued/ninfer, commit 81c8ce093b2c1646a87566a8e59d807fcf0ec95c |
| artifact | neroued/Qwen3.8-27B-nvfp4-NInfer, revision a107ba1b5b0609d9ca90d5ed61439f5f5cc7d64d |
| file | qwen3_8_27b_nvfp4.ninfer, 23.7 GB |
| sha256 | 74d2c57145e6ff11d1d2faa79594477f9bc903a611af1fb20218189fbbb77d82 |
| GPU | compute capability 12.0 (RTX 5090 or RTX PRO 6000), 32 GiB |
Use
git clone https://github.com/meshapplied/goethe.git
cd goethe/nvfp4-kernels
./install.sh && ./fetch_model.sh && ./serve.sh
| profile | drafter | draft tokens | lanes | max context |
|---|---|---|---|---|
default |
DFlash2 | 7 | 4 | 100000 |
long |
DFlash2 | 7 | 1 | 131072 |
batch |
MTP | 4 | 8 | 131072 |
All profiles: fp8 KV cache, --lm-head-draft, --prefill-chunk 1024. The server speaks the OpenAI Chat Completions,
Responses and Anthropic Messages APIs.
Performance
Decode-batch headline numbers are from the NInfer agent-replay matrix (mean context ~28k, thinking on). Extra VRAM / cold / short-random tables below use the same default/long knobs with free VRAM held to ~32 GiB on an RTX PRO 6000 Max-Q (so KV capacity matches a 5090-class card).
VRAM (idle serve, ~32 GiB free)
| NInfer process (cold, 1 lane, ctx 65k) | 24.7 GB (25 250 MiB) |
| NInfer process (warm, 4 lanes, ctx 100k) | 27.0 GB (27 600 MiB) |
| Card total (5090-class) | 32 GiB |
Agentic coding throughput
Replay of recorded mini-swe-agent turns (SWE-bench Lite trajectories in nvfp4-kernels/bench/traj/), mean context ~28k, thinking on, prefix cache on, DFlash2 K=7 + lm-head-draft patches. Headline concurrency-1 decode rate: 255 tok/s (patched stack; unpatched matrix n=1 was 239). Concurrency 4 uses the matrix aggregate at decode batch n=4. Accept length = tokens per verify round. TTFT / ITL from the same agent-replay request log (workers 1 and 4).
| Concurrency | out tok/s | accept length | TTFT ms | ITL ms |
|---|---|---|---|---|
| 1 | 255 | 4.40 | 1438 | 4.3 |
| 4 | 686 | 4.35 | 2421 | 5.4 |
Decode throughput by batch n (agent replay)
Aggregate tok/s by actual decode batch n on a physical RTX 5090 (see RESULTS.md). Same protocol as the table above; n=1 patched headline is 255.
| config | tok/round | n=1 | n=2 | n=3 | n=4 | n=5 |
|---|---|---|---|---|---|---|
| MTP, K=3 | 2.86 | 173 | 311 | 428 | 565 | 648 |
| MTP, K=4 | 3.13 | 181 | 339 | 450 | 577 | 697 |
| DFlash2, K=5 | 3.7 | 209 | 393 | 546 | 615 | - |
| DFlash2, K=7 | 4.0-4.15 | 239 | 404 | 569 | 686 | - |
| DFlash2, K=9 | 4.28 | 236 | 396 | - | - | - |
| DFlash2, K=12 | 4.37 | 236 | 405 | 525 | - | - |
Cold prefill and long-context decode
long profile, --no-prefix-reuse, 8 prompts, out=256, concurrency 1, greedy. Prefill tok/s ≈ prompt_len / mean_TTFT.
| prompt | decode tok/s | prefill tok/s | TTFT mean / p50 / p90 / p99 (ms) | ITL mean / p50 / p90 / p99 (ms) |
|---|---|---|---|---|
| 1024 | 211 | ~16k | 65 / 56 / 105 / 116 | 17.1 / 17.1 / 17.2 / 17.3 |
| 4096 | 147 | ~16k | 254 / 248 / 394 / 413 | 17.2 / 17.2 / 17.3 / 17.4 |
| 16384 | 81 | ~16k | 998 / 1010 / 1606 / 1777 | 17.4 / 17.4 / 17.7 / 17.8 |
| 32768 | 56 | ~18k | 1778 / 1586 / 3068 / 3639 | 17.6 / 17.6 / 18.0 / 18.2 |
Random prompts (short)
OpenAI chat bench, 32 prompts, in=512, out=256, prefix on, greedy (default profile). Not comparable to the long-context agent-replay matrix above.
| Concurrency | out tok/s | TTFT ms | ITL ms |
|---|---|---|---|
| 1 | 203 | 82 | 17.1 |
| 4 | 600 | 99 | 23.8 |
License
Apache-2.0. The engine and artifact are third-party and carry their own Apache-2.0 licenses and attribution (Qwen, Unsloth, z-lab, Neroued); see the NOTICE file in the goethe repository.