Qwen3.8-27B, NInfer int8-prefill artifact for the RTX 4090

The official NInfer Qwen3.8-27B artifact with one change: its prefill GEMMs may quantize their activations to int8 and run on int8 tensor cores. The weights are byte-for-byte the official ones (Q4/Q5 projections, Q8 vocabulary, MTP head, DFlash2 drafter, vision tower, indexed proposal head, tokenizer and chat template); only the activation permission of 512 projection inputs changes from A16Only to AllowA8.

This file only runs on the NInfer build from JGamboa/ninfer-4090-windows (branch main), on an NVIDIA RTX 4090 (sm_89). It does not load in llama.cpp, vLLM or Transformers, nor in engines without the int8 prefill route.

Files

qwen3_8_27b_a8_mtpq4q5.ninfer is recommended (2026-10-11): the same artifact with the MTP layer's attention and MLP projections requantized from Q8 to the Q4/Q5 mix of the text layers (mtp/input_projection stays Q8). Every other object is copied word for word. Decode is bandwidth bound, and the draft reads the MTP layer once per draft token, so a smaller layer makes each round shorter without lowering acceptance. RTX 4090, six prompts, MTP 3, --lm-head-draft, rk4v4-e8, greedy, runs alternated:

The main model is unchanged; greedy text can settle near-ties differently because the verification order changes (the Q8 artifact also does this between MTP 2, MTP 3 and no speculation). Prefill and the figures below are the same for both files.

What changes

• Prefill (129 or more tokens per step): activations are quantized per token and per 64 channels (scale = amax / 127), the same groups as the weight scales, and the Q4/Q5 codes feed m16n8k32 int8 tensor-core MMAs exactly. Each group's int32 sum is scaled by weight_scale x activation_scale in FP32.
• Decode (MTP, DFlash2 and n-gram verification included) keeps the BF16 route: the same text and speed as the official artifact.

Measurements

Current figures for this artifact (2026-09-27, main at e7d309e3, one RTX 4090 at stock clocks without a display, Windows 11, CUDA 13.4):

Against the official llama.cpp (a894dae) on the same machine, with a Q4_K_M GGUF made from the same Qwen3.8-27B BF16 checkpoint, this artifact prefills 1.75-2.0x faster and decodes 1.1x faster without speculation and 1.2-1.3x faster with MTP 3 (comparison).

Release comparison with the official artifact (2026-09-26, an earlier build, the card also driving a 4K desktop at 60 Hz). Both artifacts on the same binary, runs alternated. All figures measured.

Greedy output is identical to the official artifact whenever no prefill step reaches 129 tokens. On longer prompts some answers diverge at a later token with equivalent wording (4 of 6 test prompts, at words 3-152), as expected from a different rounding of the prompt activations.

Usage

hf download jgamboa/Qwen3.8-27B-NInfer-4090 qwen3_8_27b_a8_mtpq4q5.ninfer --local-dir E:\LLM

ninfer-serve.exe E:\LLM\qwen3_8_27b_a8_mtpq4q5.ninfer --host 127.0.0.1 --port 8080 ^
  --max-context 100000 --kv-capacity 100000 --kv-dtype rk4v4-e8 --max-concurrency 3 ^
  --prefill-chunk 1408 --spec mtp --draft-tokens 3 --lm-head-draft --ngram chain --preserve-thinking

Build instructions, every option and the benchmark commands are in the repository README.

How it was made

With the qwen3_8_27b_a8 recipe, which copies the official artifact word for word and only sets the permissions (about four minutes on the CPU, no BF16 checkpoint needed):

python -m tools.convert --model <qwen3.8 config dir> --recipe qwen3_8_27b_a8 ^
  --source reference=qwen3_8_27b.ninfer --source dflash2=<Qwen3.8-27B-DFlash2 dir> ^
  --components text,vision,mtp,dflash2 --name qwen3.8-27b --out qwen3_8_27b_a8.ninfer

qwen3_8_27b_a8_mtpq4q5.ninfer is made from qwen3_8_27b_a8.ninfer with the qwen3_8_27b_a8_mtp_q4q5 recipe (147 s on CUDA):

python -m tools.convert --model <qwen3.8 config dir> --recipe qwen3_8_27b_a8_mtp_q4q5 ^
  --source reference=qwen3_8_27b_a8.ninfer --source dflash2=<Qwen3.8-27B-DFlash2 dir> ^
  --components text,vision,mtp,dflash2 --name qwen3.8-27b --device cuda ^
  --out qwen3_8_27b_a8_mtpq4q5.ninfer

The source is the official qwen3_8_27b.ninfer from neroued/Qwen3.8-27B-NInfer (SHA-256 81f924d4...0375da). All 1,184 bound objects, the component configurations and the six resources were checked byte-identical to it.

Credits and license

Qwen3.8-27B by the Qwen team; the NInfer engine and the official artifact by Neroued; DFlash2 drafter by z-lab. The int8 prefill route was developed for the RTX 4090 in JGamboa/ninfer-4090-windows with Claude Code. Apache License 2.0, as the base artifact.