Qwen3.8-Flash-Next-Heretic-ARA-EXL3

EXL3 quantizations of Luni/Qwen3.8-Flash-Next-Heretic-ARA, a Heretic 2.0 ARA decensor of Qwen/Qwen3.8-Flash-Next (6/100 refusals vs 40/100 for the base, KL 0.191 on harmless prompts). Method and results are on the BF16 card.

Branches

Branch	Avg bpw	Weights	N-gram table	Head	MTP	Vision	Embed
`3.05bpw`	3.08	53.3 GB	39.0 GB	6 bit	4 bit	6 bit	16 bit

main holds only this card. Download a branch:

hf download Luni/Qwen3.8-Flash-Next-Heretic-ARA-EXL3 --revision 3.05bpw --local-dir ./Qwen3.8-Flash-Next-Heretic-ARA-EXL3-3.05bpw

Conversion

exllamav3 1.6.5 (dev, needs the qwen4_exp architecture), codebook mul1, default calibration (250 x 2048):

convert.py -b 3.05 -hq -hb 6 -ngb 6 -mb 4 -vb 6 -eb 16 --out_scales always

-hq gives attention, shared expert and router more bits, so the average lands at 3.08 bpw. The n-gram table (ngram_embedding.safetensors) is loaded to RAM or streamed from disk, not VRAM.

Running

TabbyAPI, 32 GB card, 128k context, routed experts partly in system RAM:

model:
  max_seq_len: 131072
  cache_mode: 8,8
  cpu_moe_split_experts: 320
  cpu_moe_pinned_arena: true
  ngram_ram: false
draft_model:
  draft_mode: mtp
  draft_num_tokens: 3

Base model: Qwen/Qwen3.8-Flash-Next
Heretic ARA BF16: Luni/Qwen3.8-Flash-Next-Heretic-ARA

Credits

Qwen: https://huggingface.co/Qwen/Qwen3.8-Flash-Next
Heretic: https://github.com/p-e-w/heretic
exllamav3: https://github.com/turboderp-org/exllamav3

Luni/Qwen3.8-Flash-Next-Heretic-ARA-EXL3