Sakura — Qwen3.8-Flash-Next, 352 of 512 experts, IQ3_XXS (base model)

GPU memory: about 31.3 GiB of weights — too big for a 32 GB card

This 3-bit build keeps about 31.3 GiB of weights on the GPU (the 26.8 GiB per-layer token table stays on disk/RAM via lazy loading). It runs on a 64 GB Strix Halo (unified memory) but does not fit a 32 GB GPU. For a 32 GB card use the 2.5-bit sibling Sakura-Qwen3.8-Flash-Next-Swift-2.5bit-352E-32GB-GGUF (about 26 GiB of GPU weights).

An experimental expert-pruned GGUF of Qwen3.8-Flash-Next (base model, no fine-tune) from ISTA-DASLab's 3-bit GSQ-RCO quantization: 352 of 512 routed experts per layer kept (chosen on German + English routing), the rest cut out without re-quantizing. Size 58.11 GiB in two shards. Made for a single 64 GB AMD Strix Halo (Radeon 8060S); German and English are the target languages. Independent community build, not an official Qwen or ISTA-DASLab release. There is also a build of the Swift 1.5 fine-tune with the same cut: Sakura-Qwen3.8-Flash-Next-Swift-3bit-352E-GGUF (fewer thinking tokens, tested more widely).

Part of the Sakura Micro line.

All Sakura Flash-Next builds side by side (what differs, measured table, runtime notes, tools): Sakura-Qwen3.8-Flash-Next-Overview.

Provenance

• Base: Qwen/Qwen3.8-Flash-Next (arch qwen4exp, 48 layers, 512 experts, 10 active).
• Quantization: ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, IQ3_XXS tier (GSQ + RCO methods, see NOTICE.md for the papers to cite).
• What this build changes: expert slicing only (see NOTICE.md). Kept experts per layer: kept_experts_K352.json.
• Expert selection: router statistics (routing mass) of this model on a German + English calibration mix; an expert is kept if it is hot in either language. Measured with llama-expert-stats, a small tool that runs the model and counts router use (much faster than llama-imatrix).

Files

SHA256SUMS lists the checksums. Load shard 1; llama.cpp finds shard 2 itself. Use -lm dio --lazy-mode on (or your build's equivalent) so the large token table is read lazily instead of held in memory.

Measured results (own tests, this exact cut)

Everything above was measured with the official llama.cpp build b11259 (Vulkan) without speculation. We did not run this cut with the MTP head or on the tasks beyond the eight anchor tasks; the Swift sibling (same architecture and size) reaches about 34 tok/s with the MTP head in a fork build, see its model card.

Held-out perplexity: llama-perplexity, context 512, 40 chunks of own German and English held-out texts (not in the calibration data), raw {"de": 2.0155, "en": 1.8087}. For orientation, same texts: the Swift 1.5 sibling at 3 bit 2.0514 / 1.8231 and at 2.5 bit 2.0681 / 1.8229.

Speed (AMD Strix Halo, 64 GB, Radeon 8060S, Windows, Vulkan)

Same ~1700-token prompt, 32k context, KV q8_0, -b 512 -ub 512 -lm dio --lazy-mode on, official llama.cpp b11259 (Vulkan): 22.7 tok/s decode, 186 tok/s prefill. ROCm is not recommended for this file: on Windows the decode collapses to about 4.9 tok/s decode as soon as the resident weights (about 31 GiB) exceed the dedicated GPU memory.

Coding check (own tasks, thinking budget 6000, max 12288 tokens, temperature 0.6, top-p 0.95, top-k 20, seeds 1000 + task number, one run each)

• Eight anchor tasks: 8/8 passed (29,345 generated tokens). The Swift 1.5 fine-tune sibling of this build needed 14,443 tokens for the same eight tasks (different runtime and budget, see its model card), i.e. about 2.0x fewer than this base-model build.

Single runs on one machine; they are indicators, not a benchmark suite. Vision was not tested (no mmproj is included; use the BF16 projector from the ISTA-DASLab repository).

Known limits

• Cutting experts without retraining costs some quality; the size of that cost was checked on German/English text and a small coding set only.
• Speculation with the shared MTP head needs a llama.cpp fork and was not measured for this cut (see above); official llama.cpp runs the file without speculation.
• Licenses: the weights inherit the Qwen Community License 1.0 (Model-as-a-Service / AI Work Assistant clause), included as LICENSE. Read it before commercial use.

Run example

llama-server -m Sakura-Qwen3.8-Flash-Next-K352-IQ3_XXS-00001-of-00002.gguf --jinja -fa on -ngl 99 -c 32768 -ctk q8_0 -ctv q8_0 -b 512 -ub 512 -lm dio --lazy-mode on --device Vulkan0

This is the command line used for the numbers above (official llama.cpp b11259, Vulkan).

---

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura — Qwen3.8-Flash-Next,512 个专家中保留 352 个,IQ3_XXS(基础模型)

GPU 内存:权重约 31.3 GiB — 对 32 GB 显卡来说太大

这个 3 比特构建在 GPU 上保留约 31.3 GiB 的权重(26.8 GiB 的逐层 token 表通过懒加载留在磁盘/内存中)。它可以在 64 GB 的 Strix Halo(统一内存)上运行,但 不能 放进 32 GB 的 GPU。对于 32 GB 的显卡,请使用 2.5 比特的同系列版本 Sakura-Qwen3.8-Flash-Next-Swift-2.5bit-352E-32GB-GGUF(GPU 权重约 26 GiB)。

这是 Qwen3.8-Flash-Next(基础模型,未微调)的一个实验性专家剪枝 GGUF,基于 ISTA-DASLab 的 3 比特 GSQ-RCO 量化: 每层 512 个路由专家中保留 352 个(依据德语 + 英语的路由情况选出),其余的被裁掉,没有重新量化。大小 58.11 GiB,分为两个分片。 为单台 64 GB 的 AMD Strix Halo(Radeon 8060S)制作;目标语言是德语和英语。这是独立的社区构建版本,不是 Qwen 或 ISTA-DASLab 的官方发布。 另外还有一个对 Swift 1.5 微调版本做了相同裁剪的构建: Sakura-Qwen3.8-Flash-Next-Swift-3bit-352E-GGUF(思考 token 更少,测试更广泛)。

属于 Sakura Micro 系列。

所有 Sakura Flash-Next 构建的并排比较(区别、实测表格、运行时说明、工具):Sakura-Qwen3.8-Flash-Next-Overview。

来源

• 基础模型:Qwen/Qwen3.8-Flash-Next(架构 qwen4exp,48 层,512 个专家,激活 10 个)。
• 量化:ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF,IQ3_XXS 档位(GSQ + RCO 方法,请参阅 NOTICE.md 了解应引用的论文)。
• 本构建的改动:仅为专家切片(见 NOTICE.md)。每层保留的专家:kept_experts_K352.json。
• 专家选择:该模型在德语 + 英语校准混合文本上的路由器统计(路由质量);只要一个专家在任一语言中是热点,就予以保留。 使用 llama-expert-stats 测量,这是一个运行模型并统计路由器使用情况的小工具(比 llama-imatrix 快得多)。

文件

SHA256SUMS 列出了校验和。加载分片 1;llama.cpp 会自己找到分片 2。请使用 -lm dio --lazy-mode on(或你所用构建中的等效选项),使巨大的 token 表被懒加载读取,而不是常驻内存。

测量结果(自己的测试,这一确切的裁剪)

以上所有数据均使用官方 llama.cpp 构建 b11259(Vulkan)在没有推测解码的情况下测得。我们 没有 用 MTP 头运行这个裁剪版本,也没有在这八个锚点任务之外的任务上运行; Swift 同系列版本(相同架构和大小)在分支构建中使用 MTP 头达到约 34 tok/s,请参阅其模型卡。

留出困惑度:llama-perplexity,上下文 512,在自己的德语和英语留出文本(不在校准数据中)上取 40 个块,原始值 {"de": 2.0155, "en": 1.8087}。 作为参考,在相同文本上:3 比特的 Swift 1.5 同系列版本为 2.0514 / 1.8231,而 2.5 比特的为 2.0681 / 1.8229。

速度(AMD Strix Halo,64 GB,Radeon 8060S,Windows,Vulkan)

相同的约 1700 个 token 的提示,32k 上下文,KV q8_0,-b 512 -ub 512 -lm dio --lazy-mode on,官方 llama.cpp b11259(Vulkan):22.7 tok/s 解码,186 tok/s 预填充。 不推荐在此文件上使用 ROCm: 在 Windows 上,一旦常驻权重(约 31 GiB)超过专用 GPU 内存,解码速度就会骤降到约 4.9 tok/s。

编程检查(自己的任务,思考预算 6000,最多 12288 个 token,temperature 0.6,top-p 0.95,top-k 20,随机种子 1000 + 任务编号,各运行一次)

• 八个锚点任务:8/8 通过(生成 29,345 个 token)。该构建的 Swift 1.5 微调同系列版本在相同的八个任务上需要 14,443 个 token(运行时和预算不同,请参阅其模型卡),即比这个基础模型构建少约 2.0 倍。

单次运行,在一台机器上;它们是指示性的,不是基准测试套件。视觉能力未经测试(未包含 mmproj;请使用 ISTA-DASLab 仓库中的 BF16 投影器)。

已知局限

• 不经再训练就裁剪专家会损失一些质量;这一损失的大小只在德语/英语文本和一个小型编程集上检查过。
• 与共享 MTP 头一起使用的推测解码需要 llama.cpp 分支,并且没有针对这一裁剪版本测量过(见上文);官方 llama.cpp 在没有推测解码的情况下运行该文件。
• 许可证:权重继承 Qwen Community License 1.0(Model-as-a-Service / AI Work Assistant 条款),以 LICENSE 随附。商业使用之前请先阅读。

运行示例

llama-server -m Sakura-Qwen3.8-Flash-Next-K352-IQ3_XXS-00001-of-00002.gguf --jinja -fa on -ngl 99 -c 32768 -ctk q8_0 -ctv q8_0 -b 512 -ub 512 -lm dio --lazy-mode on --device Vulkan0

这是用于上述数据的命令行(官方 llama.cpp b11259,Vulkan)。