Qwen3.8-Flash-Next-ROCmFPX
ROCmFPX quantized GGUF of Qwen3.8-Flash-Next (7 tiers + vision mmproj) — locally requantized from the official Qwen/Qwen3.8-Flash-Next.
License: this quantization follows the base model's Qwen Community License 1.0. It is not Apache-2.0.
⚡ Quick Start
Requires strix.cpp (this fork supports ROCmFPX; upstream llama.cpp cannot load this batch — see the Format Note at the end).
# Single machine (small tier, e.g. Q4_FAST 87.8 GiB)
strix-server -m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_FAST.gguf -ngl 99 -fa on --port 8080
# Vision: append --mmproj mmproj-Qwen3.8-Flash-Next_BF16.gguf
# Large tiers (>=113 GiB): single-machine mmap paging, or two-machine RPC pooling — see Usage
OpenAI-compatible API: http://127.0.0.1:8080/v1.
🎯 Which tier? (3-second decision)
| Use case | Recommended tier |
|---|---|
| Smallest size / single-machine full-resident | Q4_0_ROCMFP4_FAST (87.8 GiB) |
| Daily driver / long context | Q4_0_ROCMFP4_FAST_COHERENT / _STRIX (113.3 GiB) |
| Balanced quality / speed | Q6_0_ROCMFPX (135.7 GiB) |
| Agent / instruction-following | Q6_0_ROCMFPX_AGENT (158.0 GiB) |
| Quality first / maximum quality | Q8_0_ROCMFPX / _AGENT (170.2 / 173.6 GiB) |
Note: tiers above
~113 GiBcannot be fully resident on a single 128 GB machine — use mmap paging (slower) or two-machine RPC pooling (see Usage).
Model Overview
- Base model: Qwen3.8 Flash Next (official)
- Architecture:
qwen4exp— MoE, 512 experts (10 active per token), 48 transformer blocks, hidden 2560, approx. 176.9B params total - Modalities: text + vision (mmproj included: CLIP +
qwen3vl_mergervision tower) - Context: up to 256K (
qwen4exp.context_length = 262144)
Files
| File | Size |
|---|---|
| Qwen3.8-Flash-Next_Q4_0_ROCMFP4_FAST.gguf | 87.79 GiB (94,268,794,528 B) |
| Qwen3.8-Flash-Next_Q4_0_ROCMFP4_FAST_COHERENT.gguf | 113.30 GiB (121,652,681,888 B) |
| Qwen3.8-Flash-Next_Q4_0_ROCMFP4_STRIX.gguf | 113.33 GiB (121,683,156,128 B) |
| Qwen3.8-Flash-Next_Q6_0_ROCMFPX.gguf | 135.68 GiB (145,688,675,488 B) |
| Qwen3.8-Flash-Next_Q6_0_ROCMFPX_AGENT.gguf | 158.01 GiB (169,667,624,608 B) |
| Qwen3.8-Flash-Next_Q8_0_ROCMFPX.gguf | 170.15 GiB (182,698,116,768 B) |
| Qwen3.8-Flash-Next_Q8_0_ROCMFPX_AGENT.gguf | 173.58 GiB (186,375,492,768 B) |
| mmproj-Qwen3.8-Flash-Next_BF16.gguf | 0.85 GiB (907,542,784 B) |
SHA-256 checksums for all files: see sha256.txt.
Quantization Tiers
| Tier | file_type | BPW | Size | Hardening (from quantization log) | Suggested use |
|---|---|---|---|---|---|
| Q4_0_ROCMFP4_FAST | 103 | 4.26 | 87.79 GiB | none (bare baseline) | smallest footprint / single-machine full-resident |
| Q4_0_ROCMFP4_FAST_COHERENT | 104 | 5.50 | 113.30 GiB | token embedding → Q6_K; per-layer embedding → Q8_0 | daily driver / long context |
| Q4_0_ROCMFP4_STRIX | 105 | 5.50 | 113.33 GiB | + attention K/V → dual-scale FP4 | Q4 size, steadier quality |
| Q6_0_ROCMFPX | 110 | 6.59 | 135.68 GiB | uniform q6_0_rocmfpx |
balanced quality / speed |
| Q6_0_ROCMFPX_AGENT | 114 | 7.67 | 158.01 GiB | vocab/embed + selective attention → Q8_0_ROCMFPX | Agent / instruction-following |
| Q8_0_ROCMFPX | 111 | 8.26 | 170.15 GiB | uniform q8_0_rocmfpx |
quality first |
| Q8_0_ROCMFPX_AGENT | 115 | 8.43 | 173.58 GiB | vocab/embed + selective attention → standard Q8_0 | maximum quality |
Benchmarks (llama-bench)
Main table — two-machine RPC pooling (2× Strix Halo, 256 GB pool)
How it was measured: strix.cpp's RPC build (build-rpc, sha256 66dd122b…) — two Strix Halo machines (128 GB unified memory each) pooled into a 256 GB virtual machine over a 40G link. A = client, B = ggml-rpc-server.
Caliber: r5 drop-first (drop the first shot, mean of the last 4 shots — removes warm-up effects).
| Tier | config | pp256 (t/s) | pp2048 (t/s) | tg256 (t/s) |
|---|---|---|---|---|
| Q4_0_ROCMFP4_FAST | 3/1 · none | 525.89 | 565.77 | 27.99 |
| Q4_0_ROCMFP4_FAST_COHERENT | 3/1 · none | 529.56 | 552.62 | 27.84 |
| Q4_0_ROCMFP4_STRIX | 3/1 · none | 528.25 | 562.97 | 28.35 |
| Q6_0_ROCMFPX | 6/1 · dio | 311.99 | 361.93 | 22.84 |
| Q6_0_ROCMFPX_AGENT | 8/1 · dio | 363.27 | 419.87 | 21.72 |
| Q8_0_ROCMFPX | 3/1 · none | 458.33 | 530.87 | 20.22 |
| Q8_0_ROCMFPX_AGENT | 3/1 · none | 462.60 | 522.75 | 20.09 |
- "config" = layer split / load mode.
3/1: A keeps 25% of layers, B holds 75%;none/dio= the-lmvalue (rationale in the Tech Notes at the end). - r3 reference (3-shot drop-first): FAST 527.84 / 566.07 / 28.23; FC 526.26 / 555.85 / 28.06; STRIX 528.61 / 566.00 / 28.32; Q6 317.00 / 362.36 / 22.74; Q6A 364.82 / 419.17 / 21.93; Q8 464.03 / 533.08 / 20.30; Q8A 450.40 / 526.71 / 20.03.
- Sample jitter (tg256 stddev/mean) < 1% across all valid passes.
Reference only — single-machine caliber (do not compare with the table above)
| Tier | machine / caliber | pp256 (t/s) | pp2048 (t/s) | tg256 (t/s) | Note |
|---|---|---|---|---|---|
| Q4_0_ROCMFP4_FAST | B, old auto run (not mmap) | 229.65 | 331.13 | 16.25 | high variance, reference only |
| Q4_0_ROCMFP4_FAST_COHERENT | B, single-machine mmap paging | 18.37 | 32.52 | 10.47 | mixed service window |
| Q4_0_ROCMFP4_STRIX | B, single-machine mmap paging | 104.47 | 155.27 | 11.58 | clean exclusive (cleanest) |
These two groups were run differently (RPC pooling vs single-machine mmap paging) and on different engine builds — archived side by side only; do not compare the numbers across them. Single-machine paging is clearly slower than pooling (pp off by several×, tg by ~2–3×).
Usage
① Single machine (recommended for small tiers)
strix-server \
-m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_FAST.gguf \
-ngl 99 -fa on --port 8080
- Full-resident (
-ngl 99): the whole model in unified memory. The 87.8 GiB tier fits one 128 GB machine; >=113 GiB needs paging or pooling. - Paging mode (slow but runs large tiers):
-lm mmap -ngl 16. - Vision:
--mmproj <file>(no separate switch). - Context: up to 256K.
② Two-machine RPC pooling (all 7 tiers; the benchmarked method)
Steps: start the RPC server on machine B → run with --rpc on machine A (client); the model file must live on the client (A).
# ---- Machine B (RPC server) ----
systemd-run --user --no-block -p LimitMEMLOCK=infinity /bin/bash -c \
'cd ~/source/strix.cpp/build/bin && \
LD_LIBRARY_PATH=$PWD exec ./ggml-rpc-server --host 10.0.10.11 --port 50052'
# ---- Machine A (client) ----
systemd-run --user --scope -p MemoryMax=102400M -p MemorySwapMax=0 -- bash -c '
~/source/strix.cpp/build-rpc/bin/llama-bench \
-m Qwen3.8-Flash-Next_Q4_0_ROCMFP4_STRIX.gguf \
--rpc 10.0.10.11:50052 -ngl 99 --split-mode layer --tensor-split 3/1 \
-p 256,2048 -n 256 -r 5 -o json -lm none'
Key points (rationale in the Tech Notes):
- Large-embedding tiers (FAST / FC / STRIX / Q8 / Q8A):
--tensor-split 3/1+-lm none. - Q6 / Q6A:
--tensor-split 6/1(Q6A:8/1) +-lm dio. - Restart
ggml-rpc-serverbefore switching models (clears GTT); the model file must be local on the client.
Tech Notes (advanced)
Expand: load modes & RPC notes- Load mode on RPC runs: use
-lm nonefor tiers with large embeddings —-lm diobuilds an equal-size intermediate buffer that can OOM the client;-lm mmaphas a known deadlock with RPC;-lm nonereads straight into the destination. - Keep the client "light on layers": the
qwen4expembedding tensors (token_embd + per-layer + output) can only be pinned on the client's local CPU — keep the client-side layer share small. --split-mode layeris required:tensorsplit isn't implemented forqwen4exp;rowsplit isn't supported over RPC.- RPC hygiene: restart
ggml-rpc-serverbefore switching models (clears GTT); run it withLimitMEMLOCK=infinityto escape the cgroup memlock ceiling.
Format Note (ROCmFPX)
ROCmFPX is an extension format of strix.cpp (this batch uses GGML file types 103/104/105/110/111/114/115). Upstream llama.cpp cannot load them — use strix.cpp (supports all upstream GGUF formats plus ROCmFPX natively).
License & Disclaimer
- Base model: Qwen3.8 Flash Next (official) —
qwen-community-1.0(from GGUF metadata) - mmproj (vision tower): CLIP +
qwen3vl_mergerof the base model — license follows the base model - This quantized version: Apache-2.0
This is a quantized version of the official model, provided "as is", without warranty of any kind. Benchmark figures depend on the exact engine build and hardware; they are measured values under the stated environment, not end-to-end business throughput.