Expert-pruned, expert-modular repack of the Intel AutoRound W4A16 build of Qwen3.8-Flash-Next: 512 → 296 experts per layer, selected by profiling real agent traffic. Serves on a single 96 GB GPU with a 262K context window.
--expert-distribution-recorder-mode stat, producing per-(layer, expert)
route counts. A second profile was taken on anomalous contexts (the failure
sessions we wanted the model to keep handling well).experts-L{NN}.safetensors; everything else (dense, GDN linear attention, PLE,
embeddings, lm_head) into 14 large backbone shards. No tensor values were modified —
this is a pure re-chunking of the same weights, verified key-for-key identical
(129,403 tensors) with serving parity (104 tok/s).MoE layers route each token to only top-k of num_experts experts (here 10 of 512).
Routine traffic concentrates on a small subset per layer, so removing never-routed
experts is lossless for that workload; the anomaly-profile union buys back robustness
for edge contexts. Expert weights live in GPU memory only when kept, so pruning
512→296 frees ~13 GB VRAM → +31% KV pool.
| Pruned-294E | Modular (this) | |
|---|---|---|
| Experts/layer | 294 (daily profile only) | 296 (+2/layer from anomaly profile → self-heal) |
| Layout | 23 linear shards (~4.5 GB) | 48 expert files (~180 MB) + 14 backbone shards |
| Editing one layer's experts | rewrite a 4.5 GB shard + index | rewrite one 180 MB file |
| Quality gates | passed | passed (incl. anomalous-context suite) |
Same base weights, same quantization; day-to-day quality and speed are identical.
| Pattern | Count | Size each | Contents |
|---|---|---|---|
experts-L00..47.safetensors |
48 | ~180 MB | all MoE expert tensors of one layer (w13/w2 qweight/scales/qzeros) |
backbone-000XX.safetensors |
14 | ~4.4 GB | dense / GDN linear-attention / attention / PLE / embeddings / lm_head |
Swapping, adding or re-pruning experts for layer N only rewrites experts-LNN.safetensors
plus config.json (num_experts) and the index weight_map — the backbone never changes.
Want a keep set tuned to your traffic? Profile it in one command and serve pruned
without re-exporting: sglang-expert-profile
(CLI + community keep-sets; the serving-side keep-mask lives in the
sglang fork, ours/main).
--ple-offload-embedding)python -m sglang.launch_server \
--model-path ./Qwen3.8-Flash-Next-W4A16-Modular \
--chat-template ./qwen3_coder_template.jinja \
--ple-offload-embedding \
--moe-a2a-backend none \
--linear-attn-prefill-backend triton --linear-attn-decode-backend triton \
--mamba-ssm-dtype bfloat16 \
--context-length 262144 --mem-fraction-static 0.93 \
--reasoning-parser qwen3 --tool-call-parser qwen3_coder
Sampling defaults ship in generation_config.json (temp 0.7 / top_p 0.95 / top_k 20, no
penalties — Flash-Next is penalty-sensitive, see our notes in the sglang fork). No need to
pass them per request.
Note: without the marlin GC patch, loading OOMs at ~91.5 GB on some stacks (gptq_marlin_moe_repack int4→int32 expansion). Patch + details: https://github.com/ranxianglei/sglang (ours branch).
This model's 262K window + huge KV pool makes it an excellent host for our context-compression plugins — long agent sessions stay coherent while effective context grows far beyond the window:
Together: Flash-Next serves the window, billion-context compresses into it — day-long coding agents on one consumer GPU.
Base: Qwen/Qwen3.8-Flash-Next → Intel AutoRound W4A16 (calibrated) → expert pruning 512 → 296 by routing-profile keep-set → modular repack (this repo). 294-expert variant (linear shards): ranxianglei/Qwen3.8-Flash-Next-W4A16-Pruned-294E
backbone-00013.safetensors 4.7 GBbackbone-00001.safetensors 4.5 GBbackbone-00002.safetensors 4.4 GBbackbone-00003.safetensors 4.4 GBbackbone-00004.safetensors 4.4 GBbackbone-00005.safetensors 4.4 GBbackbone-00006.safetensors 4.4 GBbackbone-00007.safetensors 4.4 GBbackbone-00008.safetensors 4.4 GBbackbone-00009.safetensors 4.4 GBbackbone-00010.safetensors 4.4 GBbackbone-00011.safetensors 4.4 GBbackbone-00012.safetensors 4.4 GBbackbone-00014.safetensors 4.3 GBexperts-L10.safetensors 756 MBexperts-L11.safetensors 756 MBexperts-L12.safetensors 756 MBexperts-L13.safetensors 756 MBexperts-L14.safetensors 756 MBexperts-L15.safetensors 756 MBexperts-L16.safetensors 756 MBexperts-L17.safetensors 756 MBexperts-L18.safetensors 756 MBexperts-L19.safetensors 756 MBexperts-L20.safetensors 756 MBexperts-L21.safetensors 756 MBexperts-L22.safetensors 756 MBexperts-L23.safetensors 756 MBexperts-L24.safetensors 756 MBexperts-L25.safetensors 756 MBexperts-L26.safetensors 756 MBexperts-L27.safetensors 756 MBexperts-L28.safetensors 756 MBexperts-L29.safetensors 756 MBexperts-L30.safetensors 756 MBexperts-L31.safetensors 756 MBexperts-L32.safetensors 756 MBexperts-L33.safetensors 756 MBexperts-L34.safetensors 756 MBexperts-L35.safetensors 756 MBexperts-L36.safetensors 756 MBexperts-L37.safetensors 756 MBexperts-L38.safetensors 756 MBexperts-L39.safetensors 756 MBexperts-L40.safetensors 756 MBexperts-L41.safetensors 756 MBexperts-L42.safetensors 756 MBexperts-L43.safetensors 756 MBexperts-L44.safetensors 756 MBexperts-L45.safetensors 756 MBexperts-L46.safetensors 756 MBexperts-L47.safetensors 756 MBexperts-L00.safetensors 756 MBexperts-L01.safetensors 756 MBexperts-L02.safetensors 756 MBexperts-L03.safetensors 756 MBexperts-L04.safetensors 756 MBexperts-L05.safetensors 756 MBexperts-L06.safetensors 756 MBexperts-L07.safetensors 756 MBexperts-L08.safetensors 756 MBexperts-L09.safetensors 756 MB