Qwen3.8-Flash-Next · Avalon GGUF (IQ3_KS hybrid + MTP sidecar)

GGUF weights for Avalon, a llama.cpp fork that runs Qwen3.8-Flash-Next (512 experts, 10 active, 48 layers, ~88 GB) on a single 20 GB GPU with the experts in system RAM and a GPU expert cache.

These files only load in Avalon. They use two quantization types that stock llama.cpp does not have (IQ3_K from ik_llama.cpp, same block format; and an IQ3_KS variant with a scale per 256-weight block instead of ik's per-row scale, not file-compatible with ik_llama.cpp's IQ3_KS; both with Avalon's own type ids), a packed expert layout, and an MTP sidecar format with a cut LM head (d2t). Stock llama.cpp and ik_llama.cpp will refuse them.

Files

Main model

• Dense part (attention, routers, shared experts, embeddings, LM head): from AtomicChat's AD-4.27bpw quantization.
• Routed experts: ISTA-DASLab's GSQ-RCO quantization (IQ3_S model), with the up/gate tensors of
  • its 17 IQ3_XXS layers requantized to IQ3_KS (Avalon's variant: 3.25 bpw, a scale per 256-weight block and per-32 sub-scales), and
  • its 10 IQ3_S layers requantized to IQ3_K (3.44 bpw), both from the original BF16 weights with an importance matrix. The rest is ISTA's as is: up/gate IQ2_S (20 layers) and IQ4_XS (1 layer), down IQ4_NL (39 layers) and Q2_0 (9 layers).
• Experts packed per layer (up | gate | down contiguous), so a GPU cache fill is one copy per expert.

MTP sidecar

The official NextN/MTP head of Qwen3.8-Flash-Next, with its experts requantized (up/gate IQ3_K, down IQ4_NL, with an importance matrix), the rest Q8_0, and a cut LM head over the 65,536 most frequent tokens (FR-Spec), whose rows are copied from the main model's head. It borrows the token embeddings from the main model ("shared" head).

Quality

Perplexity, 8 × 4,096 tokens of external texts (lower is better):

MTP acceptance on a fixed text (prompt of 19k tokens, 1,023 positions, 2 draft tokens): 80.7 %.

Speed

Measured on modest hardware: RX 7900 XT 20 GB (RDNA3, PCIe 3.0 x16), Ryzen 7 3800XT (8 cores), 96 GB DDR4-3200 dual channel, ROCm 7.2. Greedy decoding, 512 tokens, median of 3 runs, MTP with 2 draft tokens:

* Maximum prefill, measured with a 32k prompt filling the 32k context. Prefill figures are without MTP (with MTP: 1,544 / 1,387 / 1,273 t/s).

Generation speed with MTP depends on how predictable the text is (acceptance 0.6 – 0.95 in these runs).

Usage

Build Avalon (see its README), then:

llama-server -m Qwen3.8-Flash-Next-Avalon-IQ3KS-00001-of-00033.gguf \
  -c 262144 -b 8192 -ub 8192 -fa on -ctk q8_0 -ctv q4_0 --jinja --parallel 1 \
  -md MTP/mtp-Qwen3.8-Flash-Next-shared-IQ3K-IQ4NL-FR65k.gguf --spec-type draft-mtp \
  --spec-draft-n-max 2 -ubd 256 -ctkd q8_0 -ctvd q8_0

--fit (on by default) places the dense weights on the GPU and the experts in RAM; the expert cache takes the rest of the VRAM. Avalon finds moe-cache-ranking.bin next to the model and fills the cache with the hottest experts before the first request (first answers of a fresh server ~5 – 8 % faster); GGML_CUDA_MOE_CACHE_PRELOAD= (empty) turns this off. The ranking was recorded with buun-llama-cpp's expert heat profile over ~18 M routings of our runs.

Credits

• Qwen: the model and its MTP head.
• ISTA-DASLab: the GSQ-RCO expert quantization (GSQ, RCO).
• AtomicChat: the AD-4.27bpw quantization (dense part).
• Unsloth: the shared BF16 MTP head we requantized.
• drluoto / Vitronia: the FR-Spec list of 65,536 tokens.
• ik_llama.cpp: the IQ3_K type and the IQ3_KS design our variant follows.
• llama.cpp and buun-llama-cpp: the base of Avalon and its expert cache; buun's expert heat profile recorded moe-cache-ranking.bin.

License

Qwen Community License 1.0, as the base model (see LICENSE).