Qwen3.6-27B, NInfer int8-prefill artifact for the RTX 4090

Qwen/Qwen3.6-27B (revision 6a9e13bd) converted from BF16 to NInfer's .ninfer format with text, vision and the MTP head in one file:

the official NInfer qwen3_6_27b representation: Q4/Q5 text layers, Q6 embedding and output head (upstream measured AIME 2026 93.3 % and GPQA-Diamond 86.9 % with it on the RTX 5090);
prefill GEMMs allowed to quantize their activations to int8 (AllowA8), as in jgamboa/Qwen3.8-27B-NInfer-4090;
the MTP layer in the same Q4/Q5 mix as the text layers (mtp/input_projection stays Q8), which shortens each speculative round on this bandwidth-bound card.

There is no DFlash2 drafter: the published one is trained for Qwen3.8.

This file only runs on the NInfer build from JGamboa/ninfer-4090-windows (branch main), on an NVIDIA RTX 4090 (sm_89). It does not load in llama.cpp, vLLM or Transformers.

Files

File	Size	SHA-256
`qwen3_6_27b_a8_mtpq4q5.ninfer`	16.1 GiB (17,317,444,864 bytes)	`2237b77d0975a8eb8e559d3df4e36b916f224845ac0602047af1a6fd2882525d`
`qwen3_6_27b_a8_mtpq4q5.ninfer.conversion.json`	0.5 MB	conversion report: sources, formats, permissions

Weights on the GPU: 15.8 GiB.

Measurements

RTX 4090 at stock clocks without a display, Windows 11, CUDA 13.4, 2026-10-11, against qwen3_8_27b_a8_mtpq4q5.ninfer on the same binary, runs alternated:

Measurement	Qwen3.6	Qwen3.8
Decode, six prompts, MTP 3, `--lm-head-draft`, greedy, thinking off	125.8 tok/s	120.9 tok/s
MTP acceptance / tokens per round	58.3 % / 2.757	56.1 % / 2.685
8K / 64K-token needle prompt, prefill (rk4v4-e8)	1.4 / 14.9 s, exact	1.4 / 14.7 s, exact
`tools.eval`, 45 deterministic tasks, thinking off	43/45	44/45
Six hard problems, thinking on, sampled, 3 seeds, up to 20,000 tokens	14/18, 204,307 tokens	18/18, 62,059 tokens
Quick perplexity, four corpora (bf16 KV)	7.1294	4.7944

Without thinking, Qwen3.6 is slightly faster per token and matches Qwen3.8 on tool calling, JSON, math, Spanish and the 4K-32K needles. With thinking it spends about 3.3x more tokens per answer, so Qwen3.8 remains the default in the release.
Perplexity differs mostly in the Chinese reference domain (26.51 against 6.78); Qwen3.6 is better on English long form (6.80 against 7.18). This is a model difference, not a uniform quantization loss.

Methods and per-problem figures are in the repository's WINDOWS_PORT.md.

Usage

Download the prebuilt Windows release and run start-qwen36-server.bat, or:

hf download jgamboa/Qwen3.6-27B-NInfer-4090 qwen3_6_27b_a8_mtpq4q5.ninfer --local-dir models

ninfer-serve.exe models\qwen3_6_27b_a8_mtpq4q5.ninfer --host 127.0.0.1 --port 8080 ^
  --max-context 100000 --kv-capacity 100000 --kv-dtype rk4v4-e8 --max-concurrency 3 ^
  --prefill-chunk 1408 --spec mtp --draft-tokens 3 --lm-head-draft --ngram chain --preserve-thinking

The API is OpenAI- and Anthropic-compatible at http://127.0.0.1:8080/v1, with a live monitor at /monitor.

How it was made

With the qwen3_6_27b_a8_mtp_q4q5 recipe from the repository (119 s on CUDA):

python -m tools.convert --model <Qwen3.6-27B BF16 dir> --recipe qwen3_6_27b_a8_mtp_q4q5 ^
  --components text,vision,mtp --resource chat_template.jinja=tools/chat_templates/qwen3_6.jinja ^
  --proposal --name qwen3.6-27b --device cuda --out qwen3_6_27b_a8_mtpq4q5.ninfer

Credits and license

Qwen3.6-27B by the Qwen team; the NInfer engine and the qwen3_6_27b representation by Neroued. The int8 prefill route and the Q4/Q5 MTP recipe were developed for the RTX 4090 in JGamboa/ninfer-4090-windows with Claude Code. Apache License 2.0, as the base model.