Qwen3.8-Flash-Next RANMA EXL3 GGUF

[!WARNING] Requires ranma.cpp ranma_20261005 or newer (github.com/saturnsky/ranma.cpp). These files use fork-specific tensor type numbers (256 and above). They will not open in upstream llama.cpp, LM Studio, Ollama or other llama.cpp-based tools.

What this is

This repository repackages the EXL3 quantizations made by turboderp of Qwen/Qwen3.8-Flash-Next into GGUF files that ranma.cpp can read.

The weights were not re-quantized. The EXL3 trellis-coded tensors are carried over bit for bit; they are only wrapped in the GGUF container (with the fork's EXL3 tensor types) and split into shards. All credit for the quantization goes to turboderp and the ExLlamaV3 project. Thank you.

Sources (EXL3, by turboderp), one branch per quantization. Each source branch maps to one folder in this repository:

Both were produced with ExLlamaV3 quantization version 1.4.4 (codebook mul1, calibration 250 rows x 2048). h5/h6 are the head bits (output layer), ng5/ng6 the n-gram embedding bits.

Not included

These files hold the text model only. Two parts of the source checkpoints are not carried over:

In llama.cpp both parts are separate files (an MTP draft GGUF and an mmproj GGUF), so they can be added later without changing the files here.

Files

Each quantization lives in its own folder. The first shard holds only metadata, the following shards hold the tensors (llama-gguf-split --split --split-max-size 50G --no-tensor-first-split). Pass only the first shard to ranma; the rest are found automatically.

File names: Qwen3.8-Flash-Next-EXL3-<bpw>-<head>-<ngram>-0000K-of-0000N.gguf. manifest.json lists the size and SHA-256 of every file.

Running

ranma.cpp keeps the routed experts of a MoE model in host RAM and reads them over PCIe from the GPU kernels. A VRAM expert cache (L1) holds the experts that the router picks most often, and a host-RAM tier (L2) bounds how much RAM the experts may use; whatever fits in neither is read from the GGUF file on the SSD. You choose L1 and L2 for your own machine. Start from the examples below and adjust.

Needed in every process: the environment variables GGML_CUDA_HOST_DIRECT=1 and GGML_CUDA_HOST_DIRECT_MAX_BATCH=512 (host-direct expert reads, HIP only; there is no CLI flag), --load-mode none on the command line, and no -ncmoe/--cpu-moe (the cache budget decides the placement). The first shard is enough for -m; the other shards are found by name.

Sizing L1 and L2

• L1. L1 = VRAM - (memory the cache does not cover) - margin. What the cache does not cover is the dense weights, the KV cache and the compute buffers; the KV cache is proportional to the context. Reference values from the benchmark runs (context 65536, -b 512 -ub 512, flash attention, f16 KV cache): total device memory was 28.4 to 29.5 GiB with L1 = 20 GiB on a 32 GiB card, i.e. about 8.4 to 9.5 GiB outside the cache, and 11.3 to 12.4 GiB with L1 = 3 GiB when emulating a 16 GiB card (again about 8.3 to 9.4 GiB outside). Keep a margin of a few GiB, more if the GPU also drives a display. The benchmark budgets were deliberately conservative; a larger L1 puts more experts in VRAM. The L1 budget includes the cache's own tables and overhead, so it is not exactly the expert bytes.
• L2. With -1, the RAM used by the experts is about the model size minus L1 (exclusive mode); the KV cache, prompt cache and buffers come on top. With a finite L2 the cache's host part is bounded by it. Set it to the RAM you can spare minus what your other programs use. The server refuses to load if the host requirement is above the physical memory installed, but it does not look at how much is free at that moment.
• If L1 + L2 covers all experts, nothing is read from the file; each expert that has to come from the SSD costs speed.

The commands and values below are examples only. Pick them for your own use and system: the context length (a long roleplay needs a large -c, a short chat or a coding session can use much less and leave VRAM to L1), the split between L1 and L2, whether to use MTP, and where the expert profile and the smart store are kept (--expert-profile-dir, --spec-smart-store; any writable location, one per model and workload works best).

Example commands (256K context)

The KV cache grows in proportion to the context. For this model it is about 30 KiB per token (f16), i.e. about 7.5 GiB at -c 262144 against about 1.9 GiB at 65536, so the L1 values below are lower than the benchmark ones. Checked on an R9700 (32 GB) with -np 1 and MTP: load and a short generation worked; dedicated VRAM peaked at 29.1 GiB (3.05 bpw) and 29.5 GiB (4.05 bpw) with L1 = 13312 (an L1 of 15360 exceeded 31 GiB and was too much). The 64 GB RAM command (finite L2 40960) was checked the same way (29.1 / 29.5 GiB). If the load fails or VRAM runs out, lower L1 by 1-2 GiB.

32 GB VRAM + 128 GB RAM (all other experts in RAM):

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h5-ng5\Qwen3.8-Flash-Next-EXL3-3.05bpw-h5-ng5-00001-of-00003.gguf ^
  -c 262144 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 13312 --expert-l2-mib -1 ^
  --expert-profile-dir .\expert-profiles ^
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf -ngld 999 --spec-type draft-mtp --spec-smart-store .\smart-store.json

32 GB VRAM + 64 GB RAM (finite L2; experts that fit in neither tier are read from the SSD):

set GGML_CUDA_HOST_DIRECT=1
set GGML_CUDA_HOST_DIRECT_MAX_BATCH=512
llama-server -m EXL3-3.05bpw-h5-ng5\Qwen3.8-Flash-Next-EXL3-3.05bpw-h5-ng5-00001-of-00003.gguf ^
  -c 262144 -ngl 999 -fa on -b 512 -ub 512 -np 1 --load-mode none ^
  --expert-cache-mode exclusive --expert-l1-mib 13312 --expert-l2-mib 40960 ^
  --expert-profile-dir .\expert-profiles ^
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf -ngld 999 --spec-type draft-mtp --spec-smart-store .\smart-store.json

The MTP lines are optional; drop them to run without speculative decoding. A 16 GB card cannot hold a 256K KV cache next to the dense weights: use -c 131072 --expert-l1-mib 2048 -np 1 there (checked on an R9700 with the 16 GB budget emulated: dedicated VRAM peak 12.9 GiB) (same --expert-l2-mib 40960 on 64 GB RAM), or a shorter context.

Benchmark settings (for the tables below)

The benchmark tables use -c 65536 -np 1 with --expert-l1-mib 20480 (32 GB VRAM), --expert-l2-mib -1 (128 GB RAM) or 40960 (64 GB RAM), and --expert-l1-mib 3072 --expert-l2-mib 40960 for the 16 GB card emulation. They are not the example commands above.

Details of every option: expert-cache.md, expert-cache-exclusive.md, expert-cache-l2.md, host-direct-moe.md, spec-smart.md, gpu-heartbeat.md.

Benchmarks

Measured on a Radeon AI PRO R9700 (PCIe 5.0 x16, headless) with ranma_20261005. The RX 9070 XT rows are emulated on the R9700 by limiting the expert cache to the budget of a 16 GiB card. Each row was measured once.

llama-bench TG128 at depth 0 (tokens/s, warm expert cache, exclusive mode):

MTP smart decoding, server decode speed (tokens/s, range over coding and four roleplay presets; draft head mtp-Qwen3.8-Flash-Next-Q8_0.gguf):

Conditions: context 65536, greedy sampling, at most 384 generated tokens for server rows, every process started after a 300 s rest. Full tables (prompt processing, deeper contexts, cold runs, memory) and the exact method: ranma.cpp benchmark document.

License

Same as the original model: Qwen Community License 1.0 (qwen-community-1.0), see the original repository. Use of these files is subject to that license.