Qwen3.8-Flash-Next-Uncensored — EXL3 2.05 bpw

EXL3 quantization of orcarouter/Qwen3.8-Flash-Next-Uncensored, an abliterated (refusal-removed) build of Qwen/Qwen3.8-Flash-Next.


Size on disk	63 GB
Language model	2.05 bpw
`lm_head`	4 bpw
MTP layers	2 bpw
Vision tower	4 bpw
n-gram (PLE) tables	4 bpw (~27 GB of the total)
VRAM, 212,992-token context, Q8 cache	~42 GB

Same recipe as turboderp's official 2.05bpw_h4_ng4 (see Conversion).

Also available: 3.05 bpw, 4.05 bpw, 5 bpw.

Disable MTP at this size

The MTP layers are quantized to 2 bpw and only ~17% of drafted tokens are accepted. On my setup, MTP on makes generation more than twice as slow (see Measurements). The MTP weights are kept in the repo for completeness, but I recommend draft_mode: disabled.

Config

model:
  model_name: Qwen3.8-Flash-Next-Uncensored-exl3-2bpw
  cache_size: 212992
  cache_mode: "8,8"
  ngram_ram: true
  tensor_parallel: false

draft_model:
  draft_mode: disabled

ngram_ram: true loads the n-gram table into system RAM instead of streaming it from disk per forward.

With autosplit and a long context, the first GPU can run out of memory during prefill. autosplit_reserve: [768, 384, 384, 96] (MB per GPU) fixed it on my 4x 20 GiB setup.

With MTP on, ExLlamaV3 1.6.x intermittently fails with operation not permitted when stream is capturing during MTP drafting. EXL3_GRAPHS=0 avoids it. With MTP off, CUDA graphs made no measurable difference on my setup.

Conversion

python convert.py \
  -i Qwen3.8-Flash-Next-Uncensored \
  -o Qwen3.8-Flash-Next-Uncensored-exl3-2bpw \
  -w /tmp/exl3-work \
  -b 2.05 -hb 4 -ngb 4 -mb 2 -vb 4 \
  --out_scales always -cb mul1 \
  -cr 250 -cc 2048 \
  -d 0,1,2,3

ExLlamaV3 1.6.3. Default calibration corpus, 250 rows x 2048 columns.

Measurements

4x RTX 4000 Ada (20 GiB, sm_89), PCIe, no NVLink, 2x Xeon Gold 5318Y. ExLlamaV3 1.6.3 + TabbyAPI, autosplit, Q8 cache (8,8), 212,992-token context. AIPerf, streaming, ignore_eos, seed 42, with warmup.

ISL 1000 / OSL 500 / concurrency 4 / 20 requests, 3 runs, 95% confidence intervals:

	Request throughput	Inter-token latency
2.05 bpw, MTP off	0.245 req/s [0.189 ; 0.302]	29.8 ms [29.4 ; 30.3]
2.05 bpw, MTP on	0.115 req/s [0.098 ; 0.132]	66.4 ms [60.4 ; 72.4]
4.05 bpw, MTP off (same bench)	0.199 req/s [0.163 ; 0.235]	37.3 ms [37.1 ; 37.5]
4.05 bpw, MTP on (same bench)	0.161 req/s [0.130 ; 0.192]	45.7 ms

2.05 bpw, MTP off, one warm run: 625 ms time to first token, 130 tok/s total output throughput (4 streams).

MTP was measured at concurrency 4 on a MoE model. Single-stream use may behave differently. Not measured.

Long context: a needle-in-a-haystack test at ~203,000 tokens returned the right answer, prefill at ~1,500 tok/s.

Quality

Not benchmarked at 2.05 bpw. Expect a noticeable loss compared to the higher-bpw versions. Abliteration figures are orcarouter's, measured on the BF16 source, not on this quant.

Safety

Safety alignment has been substantially removed via abliteration. It will comply with requests the original Qwen3.8-Flash-Next refuses, and has no meaningful built-in guardrails. Add your own moderation layer before any deployment.

License

Qwen Community License 1.0, inherited from Qwen/Qwen3.8-Flash-Next.

Lygodactylus/Qwen3.8-Flash-Next-Uncensored-exl3-2bpw