GitHub: Build-scripts and docs  |  Runtime fork  |  Discussions

Scion-FlashNext-176B-A6B: ternary MoE experts + trained corrections + 4-bit PLE

A full Qwen3.8-Flash-Next in one 57.62 GB GGUF for llama.cpp — 176B parameters in the file: the 125B language model (6B activated) plus the 51.2B-parameter n-gram table (the base's 4B MTP head is not included). Ternary expert banks plus small trained corrections, and the n-gram table (PLE) in the per-row Q4X160 container (26.24 GB) — no full-precision masters, no full-model QAT. Two larger containers ship in the same repo: the Q4_0-table file at 60.17 GB and the F16-body file at 65.80 GB.

2.60 bpw | 57.62 GB (3.2x smaller than the FP8 release) | smallest build of this model published on the Hub that we can see, pruned or not | cap_eval 15/20, 0 harness errors | HellaSwag 400 82.00%, Winogrande 400 77.50%

The name comes from grafting. A scion is the shoot grafted onto a rootstock, and here the trained corrections are grafted onto a 1.75 bpw ternary expert body.

The default file is 57.62 GB: 26.42 GB of PTQ1_0 experts, 26.24 GB of Q4X160 n-gram table, and 4.95 GB of body, corrections and headers. It is 3.2x smaller than the 185.6 GB FP8 release and smaller than every published community quant below, including the expert-pruned coder IQ1_M.

Performance

Measured with Strata, this project's own inference engine (a separate runtime from the llama.cpp fork the rest of this card documents), serving the 60.17 GB Q4_0-table file with the corrections attached at load — the local two-card box (2x RX 7900 XT, 20 GB each; layers split 26/22), the expert caches across both cards:

• ~12.4 ms per generated token at context 4096, at 2.3-2.4 tokens per pipeline window (--pipeline-windows 2, which is +62-68% over the same engine without it on this workload).
• Protocol: four requests of one fixed prompt, run 1 warms and runs 2-4 report, temperature=0, timings from the engine's own record. The embedded corrections are applied; the file is unmodified.

Measured on the local two-card box: 80.5-80.8 tok/s generation, 12.4 ms/token, 2.3-2.4 tokens per pipeline window. This is the 60.17 GB Q4_0-table file; the 57.62 GB default is not measured on this box yet.

Highlights

• One 57.62 GB file at 2.60 bpw, the smallest of three shipped containers. The official FP8 release of the same model is 185.6 GB; the published community quants are 102 GB (ggml-org IQ4_NL), 92 GB (the only other ternary build, Mooney PQ2_0), 83.6 GB (GSQ-RCO IQ3_S), 75.8 GB (IQ3_XXS), 68.0 GB (IQ2_XS) and 66.4 GB (GSQ-RCO Q2_0). This is the smallest build of this model on the Hub we can see, pruned or not: the next smallest file is ISTA's expert-pruned coder IQ1_M at 58.4 GB, and the other two Scion files are 60.17 GB (Q4_0 table) and 65.80 GB (F16 body).
• Expert banks at 1.75 bpw in the base-3 PTQ1_0 container: 5 trits per byte plus a 2-trit field and one fp16 group scale per 128 weights (28 bytes per 128). The value entropy of {-1, 0, +1} is 1.585 bpw, so the container overhead is about 10%. The body is Q6_K+Q8_0 (default variant) or F16 (fidelity variant).
• Trained, not calibrated: rank-512 correction branches on the attention outputs and the MoE block output, plus router deltas, trained by output-KD against the teacher with the deployed quantizer in the loop (ternary Lloyd g128). No imatrix and no calibration corpus, which is what separates this build from the imatrix-calibrated quants above.
• One file, no --lora: the corrections are embedded (adapter.embedded=true) and attached at load, and the n-gram table ships in the same file with per-row decoding (26.24 GB as Q4X160, 28.8 GB as Q4_0). No sidecar, no adapter plumbing, no shards.
• The gap is stated, not hidden: the 48-layer KLD gate gives mean 0.5800 (support-dominated: 92.9% of that mass is the top-512 support fit); code and logic are the weak categories (1/3 each) on the 20-task capability suite; and the step-4000 extension was trained, gated (0.5880, flat/worse) and is not shipped.
• A/B against ISTA's Q2_0, one harness: HellaSwag 400 is 82.00 vs 81.50 (a tie on the paired test) and Winogrande 400 is 77.50 vs 74.75 (a positive signal, not conclusive) at 6.2 GB smaller (8.8 GB with the Q4X160 table), while wikitext-2 PPL trails (5.457 vs 5.240) — the 1.75 bpw ternary experts buy size, not likelihood.
• First 664-task decision-arm run of this release (2026-10-09) — a floor, not a capability test: the protocol's 700/2048-token caps truncate this model's long thinking (26% of think-harder records emit no answer at all), and completed answers score 91.8% (think-harder) / 82.6% (gen). Headline floors: gen pass@1 69.4% / pass@3 80.1%, think-harder 67.6%, majority arm 71.7%; judge-head test AUC 0.862 (embeddings, cap-independent). Full truncation table under Benchmarks.

Resources

• GitHub sky-is-green/scion: the source of truth for this work — the harness that produced the file, the port decisions, the measured traps and the negative register.
• Runtime: sky-is-green/prism-ml-llama.cpp, branch qwen4exp-proto, a fork of Prism ML's llama.cpp with the PQ2_0/PTQ1_0 expert containers and the Q4X160/Q4_0 PLE tables, the qwen4exp architecture, the ffn_moe_out virtual target, embedded-adapter support and PLE table handling.
• Discussions: questions, test reports and failures are all welcome.
• Figures: the plots in this card are generated from moe/results/qwen4exp/flashnext-card-data.json by moe/plot_flashnext_card.py in the GitHub repo.

Model Overview

Weight Representation: ternary PTQ1_0 + trained corrections + compact PLE

Each expert weight takes a value from {-1, 0, +1} with one shared FP16 scale per group of 128 weights, packed base-3 in the fork's PTQ1_0 block layout (24 bytes of 5-trit data, 2 bytes holding the remaining trits, and the 2-byte group scale = 28 bytes per 128 weights = 1.75 bpw). The 51.2B-parameter n-gram table is one streamed per-row tensor decoded on the fly — Q4X160 on the smallest file (26.24 GB) or Q4_0 on the other two (28.8 GB); the correction branches and router deltas are rank-512 F16 low-rank factors merged into the file; the body carries Q6_K and Q8_0 (the F16-body variant leaves it at F16). Effective overall: 2.60 bpw (Q6_K body + Q4X160 table) and 2.72 bpw (Q6_K body + Q4_0 table).

Memory Requirement

Sizes as published on the Hub (checked 2026-10-14). Unlike the sharded community files, this release is one file with the corrections and the table inside.

Shipped Components

About the Hub's quant chip. The Hub labels a GGUF by regex-matching a known quant token in the file name (parseGGUFQuantLabel from @huggingface/tasks), not from general.file_type. The files are named after tokens the widget knows: -F16 and -Q6_K show immediately, and the shrink file is -Q6_K-Q4X160 so it labels as Q6_K today and upgrades to Q4X160 once the Hub recognizes that name (the regex keeps the last match). A PR adding Q4X160, PTQ1_0 and PQ2_0 to that list is filed. The containers themselves have no standard name: expert banks in PTQ1_0 (Prism's private ternary container, type id 143), the n-gram table in Q4X160 (146) or Q4_0, the body in Q6_K/Q8_0, norms in F32. Use the fork; stock llama.cpp knows none of these.

Shipped shrink container (Q4X160)

The PLE n-gram table (28.8 GB of the Q4_0 file) ships in a smaller container on the default file: Q4X160 - one fp16 scale per 160-element row with a 16-level codebook, 82 B/row -> 26.24 GB, file 60.16 -> 57.62 GB with the experts untouched (Q3X32, 70 B/row -> 53.76 GB, stays as the small-size fallback).

Paired code arms (164 code tasks, same pod): gen/hard deltas inside the McNemar noise band. Independent local cross-check in Strata (581 x 512 wikitext slices, paired): +0.017% PPL, not significant. The body/corrections companions (~-1.6 GB) are unbuilt. Construction: repack_experts_q2_0.py --preserve-experts --table-type q4x160 --table-bytes <prebuilt stream> (docs/PLE-TABLE-TYPES.md, docs/P4-STRATA-FIT.md).

The published artifact: Scion-FlashNext-176B-A6B-Q6_K-Q4X160.gguf, 57,615,045,408 bytes, sha256 658d56d6447cc63efdcb845ad503cd0485381f6a4aaa867973bf71753dfda0a3 (experts untouched; the table stream in it is the prebuilt work/tmp-export/q4x160-table.bin, sha256 915bae37…3015).

Q4X160 cuts the table from 28.8 GB to 26.24 GB and the file from 60.17 GB to 57.62 GB, at +0.41% PPL, -0.0001 KLD, 0.00 HellaSwag and +0.25 Winogrande. Q3X32 is 6.41 GB smaller but costs +1.30% PPL and +0.0108 KLD, so it stays a fallback and is not shipped.

Best Practices

Generation Parameters

From the base model card (thinking is on by default):

• Thinking mode: temperature=1.0, top_p=0.95, top_k=20
• Instruct / non-thinking mode: temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5

Thinking mode emits a long <think> trace before the answer; budget max_new_tokens generously (the base card works at up to 262,144 context).

Choosing Context and Offload

• Host RAM is the binding constraint, not VRAM: the 57.6 GB file is disk-backed and the experts run on CPU in the proven configuration (--cpu-moe), so plan for a host with >= 64 GB RAM. A 30 GB host thrashes the 36 GB body mapping (measured: 176 GB of reads for a 24-token completion with stock flags).
• On a 48 GB card: -ngl 99 --cpu-moe -ot per_layer_token_embd.weight=CPU keeps the PLE table and the experts host-side and the attention side on the GPU; --numa distribute tells the OS the mapping is random-access and kills the readahead amplification.
• Threads: physical cores are the right default (-t 16 on the box), and CPU-only serving (-ngl 0) is fully functional — it is how this release was measured.

Quickstart

The runtime is the fork, and the fork is the source of truth for running these files: sky-is-green/prism-ml-llama.cpp, branch qwen4exp-proto.

These files need the fork build

PTQ1_0 (expert banks), the qwen4exp architecture, the ffn_moe_out correction hook, embedded adapters and the PLE table handling (including Q4X160) all live in the fork. Stock llama.cpp will not run these files — it treats PTQ1_0 as an unknown tensor type (id 143) and Q4X160 as id 146.

# build the runtime
git clone -b qwen4exp-proto https://github.com/sky-is-green/prism-ml-llama.cpp
cd prism-ml-llama.cpp
./verify-container-support.sh          # must print "RESULT: OK"
rm -rf build
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j --target llama-cli llama-server
# ROCm: -DGGML_HIP=ON     CPU-only: no flag

If the model will not load: tensor '...' has invalid ggml type 143 (or 146) means the binary is a build of upstream llama.cpp, which knows neither PTQ1_0 (143) / Q4X160 (146) nor the qwen4exp architecture. The same error can come from a stale build/ or an older binary earlier on PATH; the reliable fix is a clean clone and build of the branch above.

# fetch the default (smallest) weights (~58 GB)
hf download SkyIsNotGreen/Scion-FlashNext-176B-A6B Scion-FlashNext-176B-A6B-Q6_K-Q4X160.gguf --local-dir .
# or the Q4_0-table file:
# hf download SkyIsNotGreen/Scion-FlashNext-176B-A6B Scion-FlashNext-176B-A6B-Q6_K.gguf --local-dir .

# CPU-only serving (the measured path; >= 64 GB host RAM)
./build/bin/llama-server -m Scion-FlashNext-176B-A6B-Q6_K-Q4X160.gguf -c 512 -t 16 --port 8080

# GPU-assist serving on a 48 GB card (experts + table stay host-side)
./build/bin/llama-server -m Scion-FlashNext-176B-A6B-Q6_K-Q4X160.gguf -ngl 99 \
    --cpu-moe -ot per_layer_token_embd.weight=CPU -fa on --numa distribute \
    -c 512 --port 8080

# chat, thinking on (base defaults)
./build/bin/llama-cli -m Scion-FlashNext-176B-A6B-Q6_K-Q4X160.gguf -ngl 99 \
    --cpu-moe -ot per_layer_token_embd.weight=CPU -fa on --numa distribute \
    -c 4096 --temp 1.0 --top-p 0.95 --top-k 20 \
    -p "Explain quantum computing in simple terms." -n 1024

Benchmarks

Community protocol (the same harness as the 35B grid): wikitext-2 PPL (-c 512 --chunks 580), HellaSwag 400 and Winogrande 400 zero-shot. Task selection is deterministic (the same 400 tasks for every model), so the multiple-choice comparison is paired; the community section below quotes the paired (McNemar) test. The KLD gate and cap_eval are the project's own harnesses.

• KLD gate: 48 layers, 8 wikitext windows x 512 tokens (4088 tokens), the student run through the same compact path the training used, the teacher parked as bf16 log-probs; full-vocabulary KLD against the bf16 teacher.
• Community PPL / HellaSwag / Winogrande: full GPU, llama-perplexity from the fork, -ngl 99 -t 8; task data passed with -f (the same hellaswag/winogrande files as the 35B grid).
• cap_eval: 20 prompt tasks (factual, math, instruction, code, logic) served by llama-server (CPU, -c 512 -t 16), 0 harness errors.
• Generation proof: the four prompts below, temperature 0, through the fork's server on ROCm/gfx1100.

48-layer KLD gate (released step-3000 vs the gated step-4000 extension)

Top-1 agreement 0.7236; teacher entropy 1.4212 nats (top-1 0.6751) vs student 1.8358 (0.6249) — the student is not sharper than the teacher. Chain-rule split of the mean: support 0.5387 (92.9%), marginal 0.0270 (4.7%), tail 0.0143 (2.5%). The KD objective sees the top-512 support (≈98.6% of the teacher mass), so the support fit is what was trained; the tail is nearly inert by design.

The released step-3000 has mean KLD 0.5800; the step-4000 extension is flat/worse at 0.5880 and is not shipped. 92.9% of the mean is the top-512 support fit, 4.7% marginal and 2.5% tail.

Community benchmark: released Q6K vs ISTA GSQ-RCO Q2_0

Measured 2026-10-06 in one A100-SXM4-80GB session (US-WA-1) with llama-perplexity from the fork (qwen4exp-proto 191929248, plus the PLE multi-sequence patch noted under Limitations), -ngl 99 -t 8, the whole model on the GPU. ISTA's file is as published; PPL is wikitext-2 context 512, 580 chunks; HellaSwag and Winogrande are 400 tasks each.

Read it as a size/quality trade: the 1.75 bpw ternary experts trail ISTA's ≈2.4 bpw GSQ-RCO experts on likelihood (≈4%), while on the two multiple-choice tasks the paired comparison over the same 400 tasks is a tie on HellaSwag (12 vs 10 discordant tasks, McNemar p = 0.83) and a positive but not conclusive edge on Winogrande (38 vs 27 discordant tasks, p = 0.22). PPL was reproduced on a second build (ours 5.4566, ISTA 5.2402) with the patch present, so the patch is not a factor in the comparison.

ISTA's published reasoning suite (AIME25 96.67, GPQA-Diamond 89.39, LiveCodeBench v6 81.14; task average 89.07, zero-shot average 78.00) is from their model card, uses a different suite, and was not measured here — it is context, not an A/B result.

Same harness, same 400 tasks. Blue = Scion, grey = ISTA; whiskers are the published CIs (PPL ±0.032/±0.033, Winogrande ±2.09/±2.18; HellaSwag has no published CI). Scion trails ISTA on PPL by 4.2%, ties on HellaSwag (paired p = 0.83), and leads on Winogrande by 2.75 points (paired p = 0.22, not conclusive). The size difference is 6.2 GB, or 8.8 GB with the Q4X160 table.

Decision-arm benchmark: 664-task reasoning/code suite (2026-10-09)

The project's community arms (experiments/cascade/community_eval.py) run on this release: 664 tasks (GSM8K-300 + MATH-200 + HumanEval-164; 465 train / 199 test), the fork/CUDA build with the embedded corrections and the Q4_0 table live, -ngl 99 (whole model on the GPU), four server slots. Gen arm: 3 samples/task, temperature 0.7, 700-token cap, thinking off. Think-harder arm: 1 sample/task, thinking on, 2048-token cap. A 40-task prefix was run end-to-end first as the harness proof under the identical protocol.

Read every number in this section as a FLOOR, not a capability measurement. This model's native thinking is long (median reasoning ~1,260 characters, roughly 2.3x the Scion-35B-A3B's on the same tasks), and the protocol's fixed caps cut it off disproportionately: on the records that actually finished, accuracy is far higher than the headline, and the headline is dominated by how often the model runs out of room.

Gen (3 samples/task)

Think-harder (thinking on, 2048 cap): all 67.6% (floor — see the truncation table below; test reasoning 71.3%, test code 69.4%, train reasoning 70.3%, train code 53.9%).

Arms (accuracy; test n=199 / all n=664): A_single 0.678 / 0.681, B_think_hard 0.709 / 0.676, C_gated 0.678 / 0.681, D_best_of_n 0.678 / 0.681, majority 0.729 / 0.717. The majority arm is the best selector on both splits; the think-harder arm helps on the test split (0.709) and not on the full set.

In-domain judge heads (last-token states through /embedding, ridge + Platt on the 465 train tasks): answer head test AUC 0.862 / balanced accuracy 0.788; trace head AUC 0.845 / BA 0.803.

Truncation at the protocol caps

The "finished" subsets skew toward shorter/easier tasks, so 82.6% / 91.8% are upper bounds and 69.4% / 67.6% are lower bounds; a fair capability number needs a larger thinking budget (the base card itself evaluates at a 256K context with long thinking budgets). The judge heads are unaffected (prompt embeddings, no generation), and the train-code think-harder drop (53.9%) is the same truncation effect at its worst.

The headline is a floor, not a capability number. "Completed" means the record finished inside the fixed token cap; it is not the model choosing to answer. On think-harder, 175 of the 178 cap-hit records emitted no final answer because the 2048-token cap stopped the thinking trace before the answer; the gen arm hit the 700-token cap on 16% of records but left no record without an answer. Scoring only records that completed inside the cap gives 82.6% / 91.8%; including cap-hit records gives the 69.4% / 67.6% floor. A fair capability number needs a larger thinking budget.

Two fork-side fixes were needed to run the suite end-to-end: the server's prompt cache disabled (--cache-ram 0, its eviction path faults with the hybrid GDN KV) and the embedding read width corrected. The measurements ran on two CUDA cards (A100-80GB for the first 1,128 samples, RTX A6000 for the rest) with identical sampling; sampling is independent per request, so the set is one statistical run.

Capability (cap_eval, 20 tasks)

Both with 0 harness errors. The Q6_K body requantises only the body (experts, table and corrections are byte-identical to the F16-body file), and the two scores are within the suite's noise band.

Q6_K scores 15/20 and F16 14/20; code and logic are 1/3 each in both. The Q6_K body requantises only the body; the experts, table and corrections are byte-identical to the F16-body file.

Generation proof (temperature 0, 32 tokens)

Use Cases

• A 125B-A6B-class model on a 48 GB card + a big host: 57.6 GB of weights in one file, experts and the PLE table host-side, attention-side layers on the GPU.
• Low-bit research and testing: a reference point for "ternary experts + trained corrections + per-row 4-bit n-gram table, no full-precision masters, no full-model QAT, no imatrix". The harness and every negative result are public.
• Local-first serving: one file, one download, embedded corrections.

Limitations

• The KLD gate is support-dominated (92.9% of the mass): distributional fidelity is a strong-2-bit/support result, not Q4-class at the tail. The gate runs 8 windows; treat small deltas as noise.
• Capability is a 20-task screen, not a benchmark suite: code and logic are 1/3 each at this checkpoint. The community PPL / HellaSwag / Winogrande numbers are in the Benchmarks section above.
• The step-4000 extension did not help (mean KLD 0.5880): the release is the step-3000 checkpoint, and the negative is recorded.
• No MTP head: the base model's 4B multi-token-prediction draft is not included, so speculative self-decoding is not available from this file.
• Text only: the base is multi-modal; no vision tensors are included.
• Host RAM: under 64 GB the mmap body thrashes; this is a serving requirement, not a model property.
• Platform coverage: CPU, ROCm (gfx1100) and CUDA (sm_80 on A100, sm_120 on RTX PRO 6000 Blackwell) verified; Metal untested.
• The fork rejects multi-sequence batches with shared tokens in the PLE path (an assert). The HellaSwag/Winogrande runs above used a local patch relaxing it; the shared tokens there are common prefixes with identical histories, so the relaxation is behavior-preserving (PPL reproduced exactly). The published fork keeps the conservative assert until the patch is upstreamed.
• Not affiliated with Prism ML, Alibaba Cloud / Qwen, or ISTA-DASLab. It builds on Prism ML's engine work (fork and containers), the Qwen base weights, and community GGUF tooling.

Citation

@misc{scionflashnext2026,
    title  = {Scion-FlashNext-176B-A6B: ternary MoE experts with trained corrections and a 4-bit n-gram table},
    author = {SkyIsNotGreen},
    year   = {2026},
    month  = {October},
    url    = {https://huggingface.co/SkyIsNotGreen/Scion-FlashNext-176B-A6B}
}

License and Attribution

Apache-2.0 for this derivative; see LICENSE and NOTICE. Base weights: Qwen / Alibaba Cloud (license:other; the FP8 release used as teacher: Qwen/Qwen3.8-Flash-Next-FP8). Container and kernels: Prism ML (MIT). Engine: llama.cpp (MIT). Comparison sizes in this card are the published files of the respective community quants (ISTA-DASLab GSQ-RCO family, ggml-org, Mooney), read from the Hub on 2026-10-14.