Qwen3.8-Flash-Next GSQ-RCO IQ3_S, NInfer v3 artifact

A NInfer v3 artifact of ISTA-DASLab's Qwen3.8-Flash-Next GSQ-RCO IQ3_S (3.50 bits per weight over the transformer weights) for NInfer-all, the master branch of iamwavecut/ninfer-all. Qwen3.8-Flash-Next is a 125B-parameter mixture of experts: 48 layers of 512 routed experts, 10 of them active per token, about 6B active parameters. The GGUF assigns a ggml quantization type to every tensor; this artifact keeps each tensor's ggml blocks byte for byte (the hyper-connection projections excepted: they are re-quantized from BF16 to Q8_0), and the engine multiplies the blocks in place. IQ3_S is the release the GSQ-RCO card recommends: the strongest of the four.

Stock NInfer builds refuse these files: the model family and the gguf_* formats exist only in that line; build master from commit 5395ec30b on. NInfer serves it with up to eight concurrent requests, prompt-prefix reuse, structured output and its Vision tower. This file includes the MTP block and incoai's DFlash2 drafter; with host experts NInfer drafts with DFlash2 unless --spec says otherwise (November 4, 2026).

What is inside

One file, Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer, of 58,242,390,528 bytes (54.24 GiB), next to its conversion report, SHA256SUMS, NOTICE and LICENSE. Original text/Vision conversion command, before the MTP and DFlash2 attachments:

python3 -m tools.convert --model Qwen3.8-Flash-Next \
  --recipe qwen3_8_flash_next_gguf --components text,vision \
  --source gguf=Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf \
  --source ngram=Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf \
  --source vision=mmproj-Qwen3.8-Flash-Next-BF16.gguf \
  --device cpu --rows-per-chunk 65536 \
  --name qwen3.8-flash-next --out Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer

The converter reads the table shard to record its digest; the rows themselves are in the table repository. The table shard of every release is the same file (SHA-256 316b46f3…e113); the digest above came from the Q2_0 release's. --model needs the configuration, tokenizer and preprocessor files of Qwen/Qwen3.8-Flash-Next (config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, preprocessor_config.json, video_preprocessor_config.json).

Running

Download the artifact and the n-gram table, then serve them from a build of master (its README; CMAKE_CUDA_ARCHITECTURES is 86, 89 or 120a, and 120a needs CUDA 13.1 or newer):

hf download WaveCut/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-NInfer-v3 \
  Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer --local-dir models
hf download WaveCut/Qwen3.8-Flash-Next-ngram-table-NInfer-v3 \
  Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer --local-dir models
M=models/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer
T=models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer

# One RTX PRO 6000 (96 GB): every expert in device memory.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --max-context 32768

# Two 32 GB GPUs (Linux): every expert in device memory, one pipeline stage per GPU.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --devices 0,1 --max-context 32768

# One 24 GB GPU: the experts in page-locked host memory (50.3 GB), the most used of them cached on the GPU.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --expert-residency host --max-context 32768

# One 24 GB GPU and little RAM: the experts stay in the file and stream into a GPU cache.
ninfer-serve $M --ngram-table $T --model-id qwen3.8-flash-next --expert-residency disk --max-context 32768

The Docker image (an NVIDIA driver of the CUDA 13 branch, 580 or newer, and the NVIDIA Container Toolkit) runs the same commands as serve with the files under /models, and answers on http://localhost:8080/v1:

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-ninfer-v3.ninfer \
  --ngram-table /models/Qwen3.8-Flash-Next-ngram-table-IQ4_NL-ninfer-v3.ninfer \
  --model-id qwen3.8-flash-next --expert-residency host --max-context 32768

With host experts the GPU holds the 3.9 GB of dense weights, the request state and an expert cache that takes what is free after startup (--expert-cache-mib sets it). With disk experts the host keeps no copy of the banks: the cache reads the experts each layer routes to from the file, through the OS page cache. The n-gram rows are read from the table's file the same way (16 per token) unless --ngram-ram loads the 28.8 GB table into RAM. A model started without its table is refused; --no-ngram-table runs it without, a non-standard experimental mode with no practical use (see the table's card). --vision adds the Vision tower (0.9 GB on the GPU) for images and video. Every expert in device memory takes 53.7 GiB on one RTX PRO 6000 and 26.5 + 27.8 GiB on two RTX 5090s; three 24 GB cards (--devices 0,1,2) should hold it too, but were not measured. The KV cache of the 12 sparse-attention layers is BF16. Qwen3.8-Flash-Next describes the placements and the execution.

November 4, 2026: DFlash2 drafter

This update appends the DFlash 2 draft model of incoai/Qwen3.8-Flash-Next-DFlash2 (Apache-2.0) as the file's dflash2 component, from nazeshinjite's Q4_0 GGUF of it (revision cbe1b23f, the smallest form of the drafter on Hugging Face): 339,417,600 bytes. Every other object is byte for byte the previous file's; dflash2-attachment.json records the sources' SHA-256 digests and the parent artifact. A round proposes up to seven drafts in one pass of the drafter, and the model verifies them, so every committed token is the model's own draw.

From commit b6e22a9c0 on, NInfer drafts with it whenever the experts are in host memory and --spec is not given: four drafts a round, verified through the first draft the drafter rates at or below 0.7 (--draft-min-p). --spec dflash2 --draft-tokens N (1 to 7) selects it explicitly, also with device experts, --spec mtp the MTP block and --spec none neither. Earlier builds load the file and ignore the drafter (checked with build a147b0c27).

On one RTX 3090 with host experts (EPYC 7K62; 256 greedy tokens after a code prompt and after a prose prompt, and the 43-token answer to a 144-token prompt; the mean of two runs) the default drafting decoded 92.3, 77.9 and 58.2 tokens per second, against 83.5, 73.4 and 49.8 with the MTP block's three drafts and the floor 0.5, and 59.0, 57.5 and 45.9 without drafts (a separate run); every answer was the answer without drafts. With four such requests at once it decoded 1.04 times the aggregate speed without drafts on the code prompt and 0.93 times on the prose prompt (MTP: 0.89 on both).

October 9, 2026: Q8_0 hyper-connection projections

The hyper-connection projections were the largest read of a decoded token, 1.27 GB in BF16. This update stores every one of them as ggml Q8_0 (32 inputs of a row share a binary16 scale): the text model's, BF16 in the GGUF, quantized with ggml's reference rounding, and the MTP block's as the Unsloth MTP GGUF stores them (this file had decoded them to BF16). The file is 618,159,104 bytes smaller; every other object is byte for byte the previous one, which the rewrite checked on readback. hc-q8-requantization.json records the transformation and its error: over the text matrices the values differ from the BF16 ones by at most 0.0334 (4.79e-4 RMS). Builds before commit 944811b5c refuse this file.

Perplexity over two 40 KB slices of the repository's corpora (int8 KV, host experts), the start of the WikiText stream of perplexity-1m and of the held-out English Wikipedia stream, moved from 1.9767 to 1.9724 (-0.22%) and from 6.5552 to 6.5604 (+0.08%). Over so few tokens one quantization moves perplexity by up to 0.7% either way; the Q2_0 build, which shares these matrices, moved by +0.015% and +0.021% over the two whole quick corpora. Earlier quality results below were measured with the BF16 projections.

Decode speed was not measured for this model; the Q2_0 build decoded 3.6-6.0% faster on one RTX 3090 with host experts, and its dense weights shrank by 0.55 GiB, which the host expert cache uses. The measurements below predate this update.

The minimum build above also fixes host experts' prompts: from commit 5118e069d (October 9) on, a prompt call of 256 tokens or more with --expert-residency host could multiply some layers by other layers' experts. Over 40 KB of WikiText the Q2_0 model read a perplexity of about 4.0 instead of 2.54 with host experts; disk and device experts were not affected.

MTP attachment and qualification

The October 8, 2026 update adds 28 MTP objects (2,791,415,296 bytes) and 1,567 bindings. Every previous text weight, Vision weight, tokenizer resource and IQ4 table descriptor passed byte-preservation checks. The separate 28.80 GB IQ4 table is unchanged.

MTP comes from the Unsloth shared-Q8_0 release, through the pinned donor recorded in mtp-attachment.json. The original text/Vision conversion report remains in the repository. The base quantization labels describe the text model, not the added MTP block.

The public Engine passed host-to-GPU and disk-to-GPU MTP requests on one RTX 3090, with Vision enabled, int8 KV, 4,096 context capacity, 128-token prefill chunks and an 8 GiB device expert cache. All expert arithmetic ran on the GPU. Each mode used two fixed prompts, three greedy repetitions and 64 generated tokens per request, with prefix reuse disabled. Each fixed mode repeated exactly and accepted MTP drafts. Image requests identified the red test image with thinking disabled. Flash-Next disables MTP for media requests and uses plain decoding. A fresh text request resumed MTP after each image check. An initial check incorrectly required MTP for media and failed. Source inspection confirmed this existing limitation; the engine behavior was not changed.

Samples include the first request. Cache contents are not reset between repetitions. Other artifacts were being prepared on the same host during this campaign, so these timings do not isolate the performance effect of MTP.

The measurements below cover only those short requests. The OS page cache was not controlled. They do not establish cold-disk speed, low-RAM operation or RTX 3080 support. Different draft widths or modes can produce different tokens because their arithmetic rounds differently. Prior quality results below were measured without this MTP attachment.

Prompt 1 requests a Python merge function after a repeated prose prefix. Prompt 2 requests a simple explanation of the blue sky. Each row summarizes the three repetitions of that prompt.

The probability-floor experiment uses the tested development snapshot. The normal MTP command above uses the default floor. mtp-qualification.json records every sample and the tested source snapshot; the minimum commit above identifies artifact support.

Quality

NInfer computes what llama.cpp computes from the same GGUF. Over the first 72 KB of the WikiText sample in the repository's eval/corpora/perplexity-1m, in 2,560-token windows that advance by 1,024 targets (llama.cpp's --ppl-stride 1024 -c 2048), the fourteen windows both programs score the same way (14,336 targets) give:

Window by window the two differ by -0.022 to +0.016 nats. NInfer applies the scales of the IQ2_S and IQ3_XXS experts in FP32 where llama.cpp's vector kernels round an integer rescale, so decode logits can differ in the last bits.

The GSQ-RCO card evaluates this quantization in llama.cpp at AIME25 100.00, GPQA-Diamond 92.93 and LiveCodeBench v6 86.86 (BF16: 100.00, 91.92, 87.43). These benchmarks have not been run in NInfer.

Speed

October 2026, greedy decoding, the repository's generate test (ninfer_qwen4_exp_generate_real) without drafts. Decode is measured over the short answer to a 20-token prompt (81 to 87 tokens) and over the first five tokens after a 4,463-token prompt; prefill is that prompt in 512-token chunks (with host or disk experts a span of eight chunks and a last chunk of 367 tokens). Host memory is the process's peak resident set; device memory is what nvidia-smi reports, and with host or disk experts the expert cache takes what is free after startup.

The single RTX 5090 and RTX 3090 rows ran on October 26, 2026, with this file, the separate table artifact and a build of NInfer-all master with CUDA 13.4, one run per row:

• RTX 5090: 575 W, PCIe 5.0 x16, a 16-core Ryzen 9 7950X, 124 GB of RAM.
• RTX 3090: 370 W, PCIe 3.0 x16, two Xeon Platinum 8272CL sockets, 1.5 TB of RAM; pinned to the CPUs of the GPU's NUMA node.

The RTX PRO 6000 (600 W, PCIe 4.0 x16) and two RTX 5090s ran this file with the separate table artifact, a 120a build with CUDA 13.1, on October 6; a range is two runs.

With host or disk experts a prompt longer than a chunk prefills in spans of up to eight chunks, so a layer's missing experts cross the host link once per span.

In early October, on an RTX 3090 host with 23 cores of an AMD EPYC 7663, llama.cpp ran this GGUF with the experts on the CPU (--n-cpu-moe 48 -t 23) at 29.6 tok/s decode (llama-bench tg128) and 312 tok/s prefill (pp512).

Credits and license

Quantized weights: ISTA-DASLab, Qwen3.8-Flash-Next GSQ-RCO GGUFs, produced with GSQ and RCO from Qwen/Qwen3.8-Flash-Next. The model is released under the Qwen Community License 1.0, reproduced in LICENSE, and its conditions apply to these files; ISTA-DASLab publish their GGUFs under Apache-2.0. Attribution notices are collected in NOTICE. This repository only re-packs those weights into NInfer's container.

Draft model: Inco AI, incoai/Qwen3.8-Flash-Next-DFlash2, in nazeshinjite's Q4_0 GGUF; both are published under Apache-2.0.