Qwen3.8-27B, NInfer int8-prefill artifact for the RTX 4090

The official NInfer Qwen3.8-27B artifact with one change: its prefill GEMMs may quantize their activations to int8 and run on int8 tensor cores. The weights are byte-for-byte the official ones (Q4/Q5 projections, Q8 vocabulary, MTP head, DFlash2 drafter, vision tower, indexed proposal head, tokenizer and chat template); only the activation permission of 512 projection inputs changes from A16Only to AllowA8.

This file only runs on the NInfer build from JGamboa/ninfer-4090-windows (branch main), on an NVIDIA RTX 4090 (sm_89). It does not load in llama.cpp, vLLM or Transformers, nor in engines without the int8 prefill route.

Files

File	Size	SHA-256
`qwen3_8_27b_a8_mtpq4q5.ninfer`	18.9 GiB (20,259,427,584 bytes)	`a52cf7f249aeb518af7c0350d7a1da0bd7c0621a3123449a767706b728927c75`
`qwen3_8_27b_a8_mtpq4q5.ninfer.conversion.json`	0.5 MB	conversion report
`qwen3_8_27b_a8.ninfer`	19.0 GiB (20,437,521,664 bytes)	`49bf76388e139defbe416c8cc0079abc1c78b59be2a41ff56ae960aeacf5fc19`
`qwen3_8_27b_a8.ninfer.conversion.json`	0.5 MB	conversion report: sources, formats, permissions

qwen3_8_27b_a8_mtpq4q5.ninfer is recommended (2026-10-11): the same artifact with the MTP layer's attention and MLP projections requantized from Q8 to the Q4/Q5 mix of the text layers (mtp/input_projection stays Q8). Every other object is copied word for word. Decode is bandwidth bound, and the draft reads the MTP layer once per draft token, so a smaller layer makes each round shorter without lowering acceptance. RTX 4090, six prompts, MTP 3, --lm-head-draft, rk4v4-e8, greedy, runs alternated:

Artifact	Decode	ms per round	Acceptance
`qwen3_8_27b_a8` (Q8 MTP layer)	117.0 tok/s	22.79	55.4 %
`qwen3_8_27b_a8_mtpq4q5`	121.0 tok/s	22.20	56.1 %

The main model is unchanged; greedy text can settle near-ties differently because the verification order changes (the Q8 artifact also does this between MTP 2, MTP 3 and no speculation). Prefill and the figures below are the same for both files.

What changes

Prefill (129 or more tokens per step): activations are quantized per token and per 64 channels (scale = amax / 127), the same groups as the weight scales, and the Q4/Q5 codes feed m16n8k32 int8 tensor-core MMAs exactly. Each group's int32 sum is scaled by weight_scale x activation_scale in FP32.
Decode (MTP, DFlash2 and n-gram verification included) keeps the BF16 route: the same text and speed as the official artifact.

Measurements

Current figures for this artifact (2026-09-27, main at e7d309e3, one RTX 4090 at stock clocks without a display, Windows 11, CUDA 13.4):

Measurement	Result
Prefill `pp512` / `pp2048` (`ninfer_bench`, int8 KV)	5,310 / 5,730-5,790 tok/s
Prefill, 64K-token prompt (needle test, rk4v4-e8 KV, answer exact)	14.8-14.9 s
Decode, no speculation (`tg128`)	54.6 tok/s
Decode, MTP 3, six mixed prompts, greedy, thinking off	120 tok/s

Against the official llama.cpp (a894dae) on the same machine, with a Q4_K_M GGUF made from the same Qwen3.8-27B BF16 checkpoint, this artifact prefills 1.75-2.0x faster and decodes 1.1x faster without speculation and 1.2-1.3x faster with MTP 3 (comparison).

Release comparison with the official artifact (2026-09-26, an earlier build, the card also driving a 4K desktop at 60 Hz). Both artifacts on the same binary, runs alternated. All figures measured.

	Official artifact	This artifact
Prefill `pp512` / `pp2048` (`ninfer_bench`, int8 KV)	2,536 / 2,762 tok/s	4,436 / 5,008 tok/s
Prefill, 8K-token prompt (needle test, rk4v4-e8 KV)	2.9 s	1.6 s
Prefill, 64K-token prompt	27.4-27.6 s	17.2 s
Prefill, 128K-token prompt	64.0 s	43.0 s
Needle-in-a-haystack answers at 8K / 64K / 128K	exact	exact
Quick perplexity, four corpora (bf16 KV)	4.800742	4.794439
Task quality, 45 deterministic tasks (`tools/eval`, thinking off)	45/45	45/45
Decode, no speculation (`tg128`)	52.0 tok/s	52.0 tok/s
MTP 3 round time	23.7 ms	23.7 ms

Greedy output is identical to the official artifact whenever no prefill step reaches 129 tokens. On longer prompts some answers diverge at a later token with equivalent wording (4 of 6 test prompts, at words 3-152), as expected from a different rounding of the prompt activations.

Usage

hf download jgamboa/Qwen3.8-27B-NInfer-4090 qwen3_8_27b_a8_mtpq4q5.ninfer --local-dir E:\LLM

ninfer-serve.exe E:\LLM\qwen3_8_27b_a8_mtpq4q5.ninfer --host 127.0.0.1 --port 8080 ^
  --max-context 100000 --kv-capacity 100000 --kv-dtype rk4v4-e8 --max-concurrency 3 ^
  --prefill-chunk 1408 --spec mtp --draft-tokens 3 --lm-head-draft --ngram chain --preserve-thinking

Build instructions, every option and the benchmark commands are in the repository README.

How it was made

With the qwen3_8_27b_a8 recipe, which copies the official artifact word for word and only sets the permissions (about four minutes on the CPU, no BF16 checkpoint needed):

python -m tools.convert --model <qwen3.8 config dir> --recipe qwen3_8_27b_a8 ^
  --source reference=qwen3_8_27b.ninfer --source dflash2=<Qwen3.8-27B-DFlash2 dir> ^
  --components text,vision,mtp,dflash2 --name qwen3.8-27b --out qwen3_8_27b_a8.ninfer

qwen3_8_27b_a8_mtpq4q5.ninfer is made from qwen3_8_27b_a8.ninfer with the qwen3_8_27b_a8_mtp_q4q5 recipe (147 s on CUDA):

python -m tools.convert --model <qwen3.8 config dir> --recipe qwen3_8_27b_a8_mtp_q4q5 ^
  --source reference=qwen3_8_27b_a8.ninfer --source dflash2=<Qwen3.8-27B-DFlash2 dir> ^
  --components text,vision,mtp,dflash2 --name qwen3.8-27b --device cuda ^
  --out qwen3_8_27b_a8_mtpq4q5.ninfer

The source is the official qwen3_8_27b.ninfer from neroued/Qwen3.8-27B-NInfer (SHA-256 81f924d4...0375da). All 1,184 bound objects, the component configurations and the six resources were checked byte-identical to it.

Credits and license

Qwen3.8-27B by the Qwen team; the NInfer engine and the official artifact by Neroued; DFlash2 drafter by z-lab. The int8 prefill route was developed for the RTX 4090 in JGamboa/ninfer-4090-windows with Claude Code. Apache License 2.0, as the base artifact.

jgamboa/Qwen3.8-27B-NInfer-4090