352 of 512 routed experts per layer: 10-11 tokens/s on a laptop with an RTX 4050 (6 GB) and 16 GB of RAM
Read the speed table before you download. The full hybrid (all 512 experts) runs on the same laptop at 9.0 tokens/s, this file at 10.8. For that difference this file gives up about a seventh of the full model's factual knowledge and loops more often in greedy code. On cards with 12 GB or more the full model is fast enough that we would take it.A pruned copy of our Hybrid GGUF of Qwen3.8-Flash-Next (IQ3_XXS trunk, Q2_0 routed experts, assembled from SC117's abliterated GSQ-RCO quants). Every layer keeps 352 of its 512 routed experts; the router still picks 10 per token. Nothing was re-quantized or retrained: the kept experts and their router rows are copied byte for byte. kept_experts_352.json lists them.
The experts were chosen for general use, not for code alone: languages, code, math and facts were all in the calibration (see How it was made).
| this file | full hybrid | |
|---|---|---|
| routed experts per layer | 352 | 512 |
| shard 1 (everything the engine keeps in memory or reads per token) | 25.8 GiB | 35.8 GiB |
| routed experts alone | 21.8 GiB | 31.6 GiB |
| shard 2 (n-gram table, one row read per token) | 28.8 GB, identical | 28.8 GB |
Decode tokens/s on Strata. Five 2000-token greedy answers, one per topic, on a freshly started engine with Strata's own expert order ("first start").
Acer Nitro V 16 (ANV16-41): Ryzen 5 8645HS, RTX 4050 Laptop 6 GB, 2x8 GB DDR5-5600, Kingston OM8PGP4512Q OEM NVMe (1.4-1.6 GiB/s in the engine's read pattern). Nobara Linux, kernel 7.2, NVIDIA driver 595. Strata 0.1.42 built for CUDA, 8K context, no MTP draft layer; the settings are under Running it.
| t/s | code | science | English prose | Cyrillic (Russian) | Chinese | mean |
|---|---|---|---|---|---|---|
| this file | 9.4 | 11.7 | 11.5 | 11.9 | 9.7 | 10.8 |
| full hybrid | 8.0 | 9.8 | 9.6 | 9.4 | 8.3 | 9.0 |
Six more code tasks (React, C#, Go, Rust, C++, Polars), 1200 tokens each:
| t/s | first start | after the engine learned the workload |
|---|---|---|
| this file | 9.3 | 13.3 |
| full hybrid | 7.3 | 10.0 |
--expert-profile-save) and load them first next time. One session of six other code tasks gave +37-43% on code here, and twice the speed on the emulated 8 GB card (below). The same profile cost 16-26% on other text, so keep one profile per kind of work. Two starters are in profiles/.--kv k8v4 --kv-resident 20480): 10.2 t/s mean, and 8.4 on the six code tasks (12.1 after learning). A 6,000-token prompt was read in 61 s; a follow-up question in the same chat started after 4 s.Our desktop (RX 7900 XTX, Ryzen 7 7700X) held to the budget: Strata limited to the given VRAM, engine and server in a cgroup with 12 GiB of RAM and no swap, 6 cores, model on a Samsung 980 PRO (2.45 GiB/s in the engine's read pattern). 32K context, MTP draft layer on.
| mean t/s | this file | full hybrid |
|---|---|---|
| 8 GB + 16 GB | 14.7 | 11.1 |
| 12 GB + 16 GB | 43.2 | 30.4 |
| 16 GB + 16 GB | 62.7 | 48.6 |
The six code tasks on 8 GB + 16 GB:
| t/s | first start | after the engine learned the workload |
|---|---|---|
| this file | 12.2 (two of the six answers looped) | 24.4 |
| full hybrid | 8.3 | 17.4 |
When the experts do not fit in VRAM plus RAM, the rest is read from the SSD for every token, and speed is roughly the SSD's read rate divided by the megabytes read per token. Pruning removes experts the router seldom picks, which the engine seldom read anyway: on the 6 GB budget this file reads 231 MiB per token and the full hybrid 273. That is why a 31% smaller expert set is only about 20% faster there.
All rows ran on the same machine with the same prompts and graders. No thinking.
| facts: NQ-open, 1800 questions | code: HumanEval + MBPP, 591 tasks | loops in 100 code answers, greedy / sampled | loops in 64 long answers, 8 languages | |
|---|---|---|---|---|
| full hybrid, 512 experts | 33.1% | 83.6% | 6 / 4 | 1 |
| this file, 352 experts | 28.4% | 80.4% | 28 / 12 | 1 |
| ISTA Coder, 256 experts | 25.2% | 86.8% | 26 / 10 | 36 |
| dense Qwen3.8-27B, GSQ-RCO IQ3_XXS | 26.9% | 83.4% | not run | not run |
| Bonsai 27B, ternary | 22.6% | 78.3% | 14 / 4 | 8 |
The Coder keeps 256 experts chosen for code and stores them at more bits, so its shard 1 is larger than this file's (27.6 against 25.8 GiB).
| this file | ISTA Coder | |
|---|---|---|
| code, 591 tasks | 80.4% | 86.8% (p < 0.001) |
| facts, 1800 questions | 28.4% | 25.2% (p = 0.001) |
| long answers that loop, of 64 | 1 | 36, in every language tested |
| 6 GB + 16 GB emulated, English prose, t/s | 13.0 | 6.8 |
| 12 GB + 16 GB emulated, English prose, t/s | 44.2 | 24.9 |
| 16 GB + 16 GB emulated, code / English prose, t/s | 71.0 / 62.9 | 43.7 / 46.1 |
The Coder writes better code. This file stays usable outside code, and on small budgets it is faster: the Coder's experts are 1.95 MiB each against 1.32, so every expert that misses memory costs more to read. On 6 and 12 GB its other speed answers looped or stopped early under greedy decoding, so prose is the one clean comparison there; on 8 GB + 16 GB its engine ran out of the 12 GiB RAM limit in our setup. On 16 GB + 16 GB the full hybrid (46.5 on code, 52.8 on prose) was also faster than the Coder. The Coder is not abliterated.
Tested with Strata only (0.1.42, commit 61b3fb5). We did not test stock llama.cpp or our Vulkan fork with this file.
Strata's releases carry Windows engines. For Linux we built the engine from source with CUDA 13 (-DSTRATA_ENABLE_CUDA=ON, as Strata's setup.py does).
# in the Strata folder
.venv/bin/python tools/iq_pack.py --gguf /path/to/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00001-of-00002.gguf \
--out packs/k352
cp /path/to/profiles/expert-profile-352E-general.bin packs/k352/
Strata's shipped data/expert-profile.bin numbers 512 experts and does not fit a pruned file; use the ones in profiles/.
strata-k352.json for 6 GB of VRAM and 16 GB of RAM:
{
"exe": "engine/strata",
"args": ["--pack", "packs/k352",
"--native", "/path/to/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00001-of-00002.gguf",
"--ple-gguf", "/path/to/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00002-of-00002.gguf",
"--expert-profile", "packs/k352/expert-profile-352E-general.bin",
"--expert-profile-save", "packs/k352/expert-profile-mine.bin", "--expert-profile-save-every", "0",
"--expert-cache", "auto", "--prefill", "256", "--spec", "4", "--spec-min-p", "0.5",
"--max-context", "8192", "--kv", "q4_0", "--mmap-experts", "--resident-budget-gib", "11"],
"cwd": ".", "tokenizer": "packs/k352/tokenizer", "model_name": "qwen3.8-flash-next-k352",
"log": "strata-k352.log", "host": "127.0.0.1", "port": 8080
}
STRATA_ROUTE_TAIL_SKIP=0 STRATA_RESIDENT_HEADROOM_GIB=2 STRATA_UNBUFFERED_LOAD=1 \
.venv/bin/python -m serve.server --engine strata --config strata-k352.json --port 8080
STRATA_ROUTE_TAIL_SKIP=0. On CUDA, Strata 0.1.42 skips the least likely experts of a missed token by default. That is about 20% faster, and on a 320-expert sibling of this file it made long Russian and Chinese answers loop. All numbers above are with it off.STRATA_RESIDENT_HEADROOM_GIB=2 with STRATA_UNBUFFERED_LOAD=1 on 16 GB of RAM. With 1.5 GiB the laptop was left with 260 MiB free. With 3 GiB and buffered reads the engine moved its file reads into the page cache and fell to 3.5 t/s."--max-context", "32768", "--kv", "k8v4", "--kv-resident", "20480".--expert-profile-save writes the session's expert order when the server stops (stop it with TERM). Point --expert-profile at that file next time."--max-context", "32768", "--kv", "int8", add "--mtp", "mtp/rt" as in the full hybrid's card, and leave STRATA_RESIDENT_HEADROOM_GIB at 2.The expert choice comes from RCO (IST-DASLab's search for the expert set that keeps the model's output distribution closest to the full model's), the method behind the ISTA Coder. We ran it on Qwen's original BF16 weights for 320 experts per layer with our own calibration: 128 sequences of 2048 tokens, the full model's answers in 16 domains at 4-7% each (text in English, other European languages, Chinese, Japanese, Korean, Cyrillic (Russian), Arabic, Hindi, Turkish and Portuguese; code, math, facts, tool calls, thinking and other). The resulting list was then applied to the abliterated GGUF. This file keeps those 320 and, in every layer, the 32 further experts the search ranked next. RCO's sets are nested in practice: the 320 set is 99.6% inside the 448 set from a separate search.
Counting how often the router picks an expert was a worse guide. Our earlier selection by routing counts kept more routing mass at 384 experts than RCO's choice (86% against 79%) and scored 5 points lower on facts (24.4% against 29.6%).
Routing profiles: expert-profile-352E-general.bin is Strata's shipped order restricted to the kept experts; expert-profile-352E-code.bin was saved by the engine on the laptop after six code tasks.
We built and measured the files below before choosing. Same tests as above; one speed session per cell.
| experts per layer | shard 1 | facts | code | code loops, greedy / sampled | long-answer loops of 64 | laptop, 6 GB + 16 GB | 8 GB + 16 GB, emulated |
|---|---|---|---|---|---|---|---|
| 320, as searched | 23.9 GiB | 26.3% | 72.9% | 46 / 18 | 1 | not run | 17.6 t/s |
| 320, with the Coder's 256 experts forced in | 23.9 GiB | 24.3% | 81.0% | 26 / 12 | 2 | 10.8 t/s | 15.5 t/s |
| 352, this file | 25.8 GiB | 28.4% | 80.4% | 28 / 12 | 1 | 10.8 t/s | 14.7 t/s |
| 384 | 27.8 GiB | 30.2% | 79.4% | 23 / 7 | 2 | 10.0 t/s | 13.7 t/s |
| 448 | 31.8 GiB | 31.3% | 85.1% | 15 / 5 | 0 | not run | 12.0 t/s |
| 512, the full hybrid | 35.8 GiB | 33.1% | 83.6% | 6 / 4 | 1 | 9.0 t/s | 11.1 t/s |
Facts fall by about 2 points for every 32 experts removed below 384 and by less than 1 point above it. Code stays at 79-81% from 384 down to 352 and drops at 320 unless the Coder's experts are kept.
The 448 set came from its own search on the same calibration. The 352 and 384 sets extend the 320 list by the search's ranking: 82% of the experts added this way are also in the 448 set, where chance would give 67%. We published only 352; the other files no longer exist.
| file | size | content |
|---|---|---|
Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00001-of-00002.gguf |
27.7 GB | IQ3_XXS trunk, 352 Q2_0 routed experts per layer |
Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00002-of-00002.gguf |
28.8 GB | per-layer n-gram embedding table, byte-identical to shard 2 of the full hybrid and of SC117's tiers |
profiles/ |
expert orders for Strata: general and code | |
kept_experts_352.json |
kept expert ids per layer, in the full model's numbering |
Vision. Stock llama.cpp and our Vulkan fork. This file under Windows. Real 8, 12 and 16 GB cards (those rows are emulated on one 24 GB card). Thinking mode. Benchmarks beyond the four above. Each speed row is one session; repeated Strata runs of the full model varied by up to 8%.
LICENSE).Not affiliated with any of them. The model is abliterated: its refusals were removed upstream, and you are responsible for how you use it.
Qwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00002-of-00002.gguf 29 GBQwen3.8-Flash-Next-GSQ-RCO-abliterated-IQ3_XXS-Q2_0-352E-00001-of-00002.gguf 28 GBprofiles/expert-profile-352E-code.bin 135.2 kBprofiles/expert-profile-352E-general.bin 67.6 kB