Qwen3.8-27B for NInfer on the RTX 3090

Maintained at ashalliants/ninfer-3090: the engine, converter and release builds these artifacts are made and tested with. Releases, issues and updates live there, and a star helps others find it.

This model card is the version-controlled source for WarlaxZ/Qwen3.8-27B-NInfer-3090.

The repository contains Qwen3.8-27B converted to the native .ninfer artifact format by the RTX 3090 fork of NInfer, ashalliants/ninfer-3090. It is the fork's default 27B model (download-model qwen38-27b). It is not a Transformers checkpoint, Safetensors distribution or GGUF file, and it needs this fork's executables: upstream NInfer and older fork builds refuse it at startup because they lack its 4-bit MLP down route.

What is different from the upstream artifact

Same model, same container, a different weight encoding. Upstream's neroued/Qwen3.8-27B-NInfer rounds each 64-weight group to the nearest code with scale = absmax / qmax. This artifact uses the grouped_search encoder:

• each group's FP16 scale is searched over 50 candidates, choosing the one with the least importance-weighted squared error, where importance is the mean squared input activation per channel (an imatrix);
• scales may be negative, which lets a group use the full code range (−8 for Q4, −16 for Q5);
• the layout spends fewer bytes per decoded token: a 4-bit token embedding, a 6-bit output head, and 4-bit attention/GDN output projections and MLP down projections in layers 36–63. Everything else keeps upstream's formats (Q4 query/key/gate/up, Q5 value/z/gate/down/output).

Vision, MTP, DFlash2 and the 131,072-row proposal head are carried over with upstream's encoding.

Artifact

printf '%s  %s\n' \
  '7f2a0086a071ee932c66106e1552d118a85cfec8ddef13c67a54cd64abb39706' \
  'qwen3_8_27b.ninfer' | sha256sum --check

The repository also holds qwen3_8_27b.ninfer.conversion.json (the converter's report: every object, its sources, format and method) and qwen3_8_27b.imatrix.safetensors, the importance matrix the encoder used, so the artifact can be rebuilt.

Measurements

RTX 3090 (24 GB), CUDA 12.8, Windows. "Upstream" is neroued/Qwen3.8-27B-NInfer at 1cbd84e7, run with the same binaries. Quality was measured at fork commit 66a05b72; speed with the small-T Q6 head kernel that followed it (36b798c9), which this artifact needs to be faster under speculative decoding.

Quality. ninfer-perplexity --quick --kv-dtype int8:

KL divergence against Unsloth's Q8_0 GGUF on the held-out corpus (8 × 4096 tokens per stream, llama.cpp b11316 with the NInfer weights exported bit-exactly into Q8_0 blocks):

The 4-bit embedding costs Chinese Wikipedia (0.0794 → 0.0859) and chat (0.0355 → 0.0373) against a 6-bit one; English, arXiv and code streams are unchanged to the fourth decimal.

Unsloth's UD-Q4_K_XL is still clearly better per byte. Their formats are not NInfer's, and about half of the gap is their per-tensor allocation, which needs kernel routes NInfer does not have yet.

Speed. ninfer_bench, rk4v4 KV, 8,192-token context, two interleaved rounds of three repetitions each:

The bench corpus is unusually easy to draft (DFlash2 accepts 7.3 tokens per round on it), so the speculative rows were also measured on 24 varied chat prompts through ninfer-serve (greedy, thinking off, 512 tokens each, one stream):

MTP acceptance is about 1-1.5% lower than upstream's, so its gain is smaller than DFlash2's. Before the small-T Q6 kernel, the 6-bit head's 8-token verify took 2.56 ms against 1.58 ms for upstream's 8-bit head and cancelled DFlash2's gain (−0.6% on the bench); with it the head reads at about 850 GB/s from 3 to 8 tokens.

Against llama.cpp. Decode speed on one RTX 3090, one stream, greedy, 8K context. The llama.cpp figures are not ours: they are Hsiu-Chi Tsai's published measurements (2026-08-29, Debian, i9-13900, Unsloth UD-Q4_K_XL, 25 prompts times 5 passes, best draft count per mode). Ours are the rows above, on a different machine and a different prompt set, so read the ratios as indicative rather than exact.

Unsloth's file is closer to Q8_0 than this artifact (KL divergence 0.0156 against 0.0318), so this is a speed comparison, not one at equal quality. The same post notes llama.cpp's MTP output did not match serial greedy decoding byte for byte.

Memory. Started back to back through run.bat qwen38-27b on the same desktop. The earlier run.bat transcoded upstream's embedding to 4 bits at load, so the saving is smaller than the file size difference:

With the desktop holding 1.6 GiB, the DFlash2 profile starts at 188,416 tokens with 721 MiB free and is refused at 196,608, so run.bat now defaults to 188,416. run.bat no longer passes --embedding-q4 or --lm-head-q6, which this artifact does not need; the upstream file therefore no longer fits the default profiles.

Smoke tests. These all completed coherently:

• both run.bat profiles served a chat request;
• DFlash2, MTP3 and plain text generation;
• image_chart (NIFER VISION 731；3；左侧, the expected answer);
• image_natural (mailbox 24, sun on the right) with --vision --vision-residency overlay.

Run it

Build ashalliants/ninfer-3090, then:

scripts\download-model.bat qwen38-27b
scripts\run.bat qwen38-27b

or on Linux scripts/download-model.sh qwen38-27b and scripts/run.sh qwen38-27b. Direct CLI use:

./build/apps/ninfer models/qwen3_8_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 32768 --kv-dtype rk4v4 \
  --spec dflash2 --draft-tokens 7 --lm-head-draft

DFlash2 needs --lm-head-draft. Its candidate top-k reads either the proposal head or the full head, and the top-k kernel has no 6-bit route, so DFlash2 with the full 6-bit head is refused. --embedding-q4 and --lm-head-q6 are accepted and do nothing here, because both tensors are already stored in those formats.

Provenance

artifact-manifest.json has the object inventory and every source hash.

Limits

• Vision, MTP, DFlash2 and proposal-head weights use upstream's round-to-nearest encoding. The imatrix covers the text layers only, and the embedding and head are searched without weights.
• The imatrix is Unsloth's general calibration set, not one tuned for this engine's workloads.
• NInfer executes on one CUDA device, or splits layers across up to eight GPUs on Linux. It does not do CPU offload, and it does not execute generated tool calls.

License

Apache License 2.0, matching Qwen3.8-27B, the DFlash2 companion weights and Unsloth's importance matrix.

---

Built and maintained by ashalliants/ninfer-3090. Development is sponsored by NeverMetered, hosted Qwen3.8-27B for coding agents.