Qwen3.8-Flash-Next in the light DS4-IQ2 packaging of ivanfioravanti/Qwen3.8-Flash-Next-DS4-IQ2: an IQ2 main (IQ2_XXS gate/up, Q2_K down padded to 768, embedded MTP block) plus an external demand-paged PLE Q4_1 sidecar instead of a ~95 GiB resident BF16 n-gram. Runs resident and zero-swap on a 64 GiB Apple Silicon box (measured on M5 Pro), 8K→220K context.
| model | MTP-off | single-stream --mtp |
notes |
|---|---|---|---|
| Q8 dense (exact) | ~30–32 t/s | ~38 t/s | most accurate |
| Q4_K imat dense | ~36 t/s | ~42–43 t/s | +~14% vs Q8, near-Q8 quality |
--mtp) adds ~+17% single-stream over MTP-off; draft acceptance ~67%.DS4_QWEN4_MTP_DEPTH, default auto): drafts a
2nd token on deterministic/structured/code continuations for a further +5–6%,
and falls back on free-form prose so it never regresses. Output stays
autoregressive-exact (verify only commits argmax-matching drafts → zero quality
change).--batched-session N, concurrent requests,
full quality): ~67 t/s @ 8 streams, ~80 t/s @ 16 streams aggregate (~2× a
single stream). Decode is memory-bandwidth-bound on this hardware, so aggregate
throughput plateaus around ~80 t/s rather than scaling linearly; a single stream
cannot exceed ~43 t/s here (that ceiling needs a smaller model or more MTP heads,
not tuning). Concurrency is bounded by ctx × sessions KV vs 64 GiB (ctx 4096 fits
16 streams; large ctx needs fewer).The IQ2 experts are identical across both; they differ only in how the per-layer dense projections are quantized:
| main | dense | size | decode vs Q8 | logit cosine vs Q8 | pick when |
|---|---|---|---|---|---|
...-Q2KDownPad768-MTP.gguf |
Q8_0 | 41.73 GiB | baseline | 1.000 (exact) | most accurate output |
...-DenseQ4Kselimat-MTP.gguf |
selective Q4_K + imatrix | 40.68 GiB | +~14% | 0.949 (argmax preserved) | faster decode, near-Q8 quality |
The DenseQ4Kselimat variant requantizes the less-sensitive dense projections
Q8_0→Q4_K (routed to DS4's fast dense Q4_K GEMV) while keeping the most
quality-sensitive full-attention q/k/v/output projections at Q8_0, and weights
each dense tensor's Q4_K quantization by an importance matrix (activation energy
per input column). Net: ~+14% decode at logit cosine ~0.949 vs Q8 (argmax preserved,
coherent). Encode-only change (same runtime kernel/bytes as plain Q4_K; the imatrix
costs nothing at inference). imatrix source: the published llama.cpp calibration
unsloth/Qwen3.8-Flash-Next-GGUF (same qwen4exp tensor decomposition).
Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-MTP.gguf — Q8 dense main
(exact). 44,806,612,448 bytes, sha256
e078c60abfdc9c5dd849eddb41660e5fc8f1b0da2f1200c643a8d3ec50324e8a.Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-DenseQ4Kselimat-MTP.gguf —
imatrix-weighted selective-Q4K dense main (faster, near-Q8). 43,677,591,168
bytes, sha256 b1dd08509231126b5f1596603fd2589ff8bdf8eae221420f3e4409110fee8bd4.Qwen3.8-Flash-Next-PLE-Q4_1.gguf — PLE Q4_1 n-gram sidecar, shared by both
mains (required). Byte-identical between base Qwen and this fine-tune
(abliteration does not touch the n-gram table), so it is Ivan's sidecar verbatim.Qwen3.8-Flash-Next-PLE-Q4_0.gguf — experimental PLE sidecar requantized Q4_1→Q4_0 with
gguf-tools/ple_requant.py (−3 GiB). Not recommended: round-trip relL2 0.138 vs
Q4_1, and measured no speed or context gain (PLE is CPU-mmap, not Metal-wired).
Drop-in for --ple. 28,800,141,632 bytes, sha256
68920ba4887e7e463263ce0c771aacd1902b315c89145ab090e08e3c2d69d611.Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-SelQ4Down18-48-DenseQ4Kselimat-MTP.gguf —
"unc31" selective tier: 30 of 48 down layers (18–47)
Q4_K. 51,201,124,096 bytes, sha256
7f45767af63a45b17971870d2288d8dbdcd92236f4961c69f531df165aaadef8.Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownPad768-DenseQ4Kselimat-MTP.gguf —
"unc48L" (Complete), strongest uncensor: all 48 trunk down layers Q4_K, MTP
blk.48 down Q8_0. Needs SSD expert streaming on 64 GiB (see "Running unc48L at 256K").
55,447,856,768 bytes, sha256
20b4c617a5e7a8d3976d328daf04c48c0f250f47c9a1e7ad91265e698616c41d.Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownTrim-DenseQ4Kselimat-MTP.gguf —
"unc48L Trim": the same model as unc48L with the padding removed from its Q4_K down rows.
A down row holds 640 real values, and unc48L pads it to 768 to fill whole 256-value Q4_K blocks.
The Trim file keeps each row's two whole blocks plus the header and real nibbles of the third
(368 bytes instead of 432). Every byte the kernels read is unchanged, and the output is
byte-identical to unc48L. 3.75 GiB smaller. Needs ds4-metal develop 331cf3b2 or later;
older engines refuse the file. 51,421,324,928 bytes, sha256
3a7c8430b25d3e1a3feeaa7c620e642d453f4d41f9a8c3d9d86d1e94afef22b9. The .gguf.json next to it
lists every tensor's sha256.adapters/refusal-4-44.f32 — optional runtime refusal-projection adapter (480 KiB).
See "Runtime refusal projection" below.Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt — optional MTP draft vocabulary
(65,568 token ids, most frequent first; 390,414 bytes, sha256
f8b0dc7c28f173c4b071c986cf44d59ff4f169ad5a6442a88ea240e3b1fbe652). Shared by every main; see "MTP draft vocabulary" below.Uncensor strength is set by how many trunk down-expert layers survive the abliteration delta — 2-bit (Q2_K) down experts erase most of it, Q4_K preserves it. The tiers:
| tier | Q4_K down layers | uncensor strength (10-prompt probe, greedy) | fits on M5 Pro 64 GiB |
|---|---|---|---|
| Q2K main | 0 | weakest: only low-band relaxes (profanity) | 220K–256K |
| unc8 (removed 2026-10-03) | 7 (41–47) | ≈ Q2K baseline: mid-band (phishing/hotwire) still refused | 256K, ~42 t/s @4K |
| unc16 (removed 2026-10-03) | 15 (33–47) | ≈ Q2K baseline: mid-band still refused | 256K, 40.1 t/s @4K |
| unc31 | 30 (18–47) | mid-band full: phishing/hotwire/lockpick comply; break-in + extreme (meth/IED) still refused | 256K with SSD streaming (K=36): 46.5 t/s short prompt |
| unc48L (Complete) | 48 (0–47) | strongest (extreme-band, disclaimer-led) | 256K with SSD streaming (K=32): ~42–43 t/s short prompt, ~34 t/s prose at 256K full, 49.3 GiB peak |
Pick unc48L Trim (or unc48L on an older engine) for the strongest uncensor, unc31 for mid-band uncensor with a little more speed and faster prefill, or the Q2K mains for the lightest footprint. The unc8 and unc16 files were removed on 2026-10-03 because they refused like the Q2K mains. unc31 and unc48L at long context need the SSD-streaming engine below; the older engine OOMs them at 256K.
The ds4-metal develop engine keeps the first K routed layers resident and streams the
experts of the rest through a locked RAM cache; KV is 4-bit by default. On M5 Pro 64 GiB:
DS4_QWEN4_STREAM_FULL_LAYERS=32 DS4_QWEN4_PLE_PREFETCH_FULL=0 \
DS4_QWEN4_MTP_DRAFT_VOCAB=Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt \
ds4-server -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q4KDownPad768-DenseQ4Kselimat-MTP.gguf \
--ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf -c 262144 --prefill-chunk 2048 --mtp \
--ssd-streaming --ssd-streaming-cache-experts 6GB
Measured (2026-09-23, greedy, MTP + 64K draft vocab, server warm; peak = system-wide wired):
| unc48L config | short prompt VI/EN/code | full context, prose answer | full context, counting (MTP-friendly) | peak wired | expert cache |
|---|---|---|---|---|---|
| K=36, 6 GB | 43.5 / 41.9 / 43.6 t/s | — | 37.8 @226K | 52.6 GiB | capped at 0.22 GiB (over the wire limit) |
| K=32, 6 GB | 42.9 / 41.6 / 43.0 t/s | 34.1 @256K | 43.8 @226K, 43.1 @256K | 48.9–49.3 GiB | fully locked |
| K=32, 8 GB | +2% vs 6 GB | 34.3 @256K | 43.7 @226K | 50.9–51.4 GiB | fully locked |
| K=30, 6 GB | ≈ K=32 | — | 38.4 @256K | 49.0 GiB | fully locked |
The counting column (answer + "count from 1 to 400") is what MTP predicts best, so it overstates real use. Decode speed is nearly flat with context length: the same reasoning-heavy prose task (summarize a C file, thinking on) runs 34.4 / 35.1 / 36.0 / 34.1 t/s at 37K / 71K / 135K / 256K (unc31: 35.9 at 135K). Simple short chats run ~42 t/s. What changes the speed is the kind of answer (how well MTP predicts it), not the context length.
unc48L Trim runs with the same command; only -m changes, and it needs ds4-metal develop
331cf3b2 or later. Measured on 2026-10-03 against unc48L (M5 Pro 64 GiB, the command above):
Use K=32. Prefill at 256K runs ~310 t/s (about 14 min for a full 256K prompt); unc31 prefills ~30% faster. Needle retrieval passed in every full-context run, swap stayed flat.
The MTP draft only needs its argmax, so it can be scored over the most frequent
output rows instead of all 248K. DS4_QWEN4_MTP_DRAFT_VOCAB=<file> gathers those rows
into a small draft head (~178 MiB); the verify pass still uses the full head, so every
committed token is still checked by the full model. The Qwen3.8 tokenizer puts
Vietnamese at high token ids, so a plain id prefix (DS4_QWEN4_MTP_DRAFT_ROWS)
collapses Vietnamese draft acceptance (71% -> 36%); this list ranks ids by measured
frequency over Vietnamese and English Wikipedia, code and Python instead, and covers
100% / 99.4% / 97.0% of the model's own Vietnamese / English / code output.
Measured on unc31 (M5 Pro 64 GiB, 8K, greedy, 500 tokens, two runs each):
| prompt | full head | this list |
|---|---|---|
| Vietnamese | 35.5 t/s (71.0% accept) | 36.7 t/s (71.0%) |
| English prose | 35.3 t/s (68.3%) | 36.9 t/s (68.3%) |
| code | 36.1 t/s (72.7%) | 37.5 t/s (70.3%) |
DS4_QWEN4_MTP_DRAFT_VOCAB=Qwen3.8-Flash-Next-draft-vocab-vi-en-code-64k.txt ds4 -m ... --mtp ...
Built with gguf-tools/frspec_vocab.py (ds4-metal fork, branch develop).
adapters/refusal-4-44.f32 removes the refusal direction from the residual stream at run time:
h -= s·(h·v)·v after each FFN output, one unit direction v per layer, on layers 4–44. No weights
change, so it also works on the stock (non-abliterated) weights.
Qwen3.8-Flash-Next-refusal-projection.gguf), converted to ds4's per-layer
f32 format by dir-steering/tools/cvec_to_f32.py --layers 4-44. adapters/refusal-4-44.f32.json
holds the provenance and both sha256 sums.ds4-server -m <main>.gguf --ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf ... \
--dir-steering-file adapters/refusal-4-44.f32 --dir-steering-ffn 0.5
Measured with ds4 on the stock Ivan DS4-IQ2 weights, at scale 0.5 (frozen eval harness, 2026-09-29/30):
The source repo reports 1/50 harmful refusals at KL 0.186 for the full-strength projection in llama.cpp.
Runs on the ds4-metal engine (llama.cpp-lineage Metal fork for qwen4exp). The
build used here adds, over the stock light runtime: the external --ple sidecar
loader combined with native multi-session batched decode, the fast Q4_K dense
GEMV, adaptive MTP draft depth, and correctness fixes (accurate GDN-gate
softplus, an isfinite guard in the expert reduction, ignore_eos honored under
--mtp, and a clearer tool-call parse error). The --ple sidecar is CPU-mmap
demand-paged (each token faults ~one page per hash head); it fits 64 GiB beside the
resident model, whereas the embedded-BF16 n-gram lineage does not.
# exact (default)
ds4 -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-MTP.gguf \
--ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf --mtp -c 4096 -p "..."
# faster (imatrix-weighted selective dense Q4_K)
ds4 -m Qwen3.8-Flash-Next-OrcaUncensored-IQ2XXS-Q2KDownPad768-DenseQ4Kselimat-MTP.gguf \
--ple Qwen3.8-Flash-Next-PLE-Q4_1.gguf --mtp -c 4096 -p "..."
# serving many concurrent requests (ds4-server, ~2x aggregate)
ds4-server -m ...DenseQ4Kselimat-MTP.gguf --ple ...PLE-Q4_1.gguf --mtp \
--batched-session 16 -c 4096 --host 127.0.0.1 --port 8000
Long contexts: add --prefill-chunk 2048 to stay zero-swap at 128K–220K on 64 GiB.
Uncensored weights from orcarouter/Qwen3.8-Flash-Next-Uncensored
(rev 8336e613ea508b13c2159bd0f68965d97a606b95), quantized to Ivan's DS4-IQ2 layout,
then repackaged into this light form by stripping the embedded n-gram tensor
(retained tensors byte-identical; external Q4_1 sidecar substituted). The
DenseQ4Kselimat main is derived from the Q8 main by imatrix-weighted Q4_K requant of
the dense projections (attn q/k/v/output kept Q8_0).
This is an abliterated fine-tune; the 2-bit light-quant dilutes most of the abliteration, so the base Q2K mains behave much closer to the original base model than to a full abliteration (they relax mild refusals but keep most harmful refusals). The SelQ4 tiers progressively restore the abliteration delta on the down experts (table above): unc31 is the first tier where the mid-band requests comply; the extreme categories (drug/weapon synthesis, IED) and break-in stay refused on every tier except unc48L (Complete), which only reaches them with a disclaimer. Treat every tier as a research artifact, not an unrestricted model. Provided for research; you are responsible for lawful use.