Qwen3.8-Flash-Next-exl3-3.05bpw-heretic

This is an abliterated build of turboderp/Qwen3.8-Flash-Next-exl3, revision 3.05bpw_h5_ng5. It was made with Heretic running directly on the quantized EXL3 model.

The EXL3 weights are bit-identical to the original. Nothing was requantized or merged. The abliteration lives in a small sidecar, abliteration.safetensors (0.7 MB). At inference time it is applied exactly to the outputs of the unmodified quantized layers. Every model-*.safetensors, ngram_embedding.safetensors and mtp_hyper_connection_mixer_patch.safetensors file is the same file as in the base repository.

The model

Qwen3.8-Flash-Next is a 125B-parameter MoE with about 6B active parameters per token:

• 48 layers: 36 Gated DeltaNet and 12 Qwen Sparse Attention.
• 512 experts per layer, 10 routed per token, plus a shared expert.
• Four hyper-connection residual streams.
• A 51B-parameter hashed n-gram embedding table.
• One MTP layer for speculative drafting.

This build is turboderp's 3.05 bpw EXL3 quantization, with a 5-bit head, 5-bit n-gram table and 5-bit vision tower. It totals ~80 GB: 47 GB of experts, the 33 GB n-gram table and ~3 GB for everything else.

How the abliteration works

Heretic removes a "refusal direction" v from what each layer writes into the residual stream. Every ablated sublayer output y is modified in place:

y  <-  y - λ · v · <y, v>

The ablated sublayer outputs are:

• the attention / Gated DeltaNet output projection;
• the MoE block output, meaning the sum of the routed experts and the shared expert.

The formula is linear in y, so applying it to the summed MoE output is exact. That matters because there are 512 experts per layer, and with CPU offload most of them run on the CPU. The vectors are stored per layer in abliteration.safetensors as <block>.abl_attn.{a,b} and <block>.abl_mlp.{a,b} (a = v, b = λ·v). Their λ already includes the factor described next.

Hyper-connections need a reflection, not a projection

On a standard transformer, λ ≈ 1 is enough. It projects v out of each sublayer's contribution to the single residual stream. On this model it was not:

• A first study over Heretic's normal range (λ ≤ 1.5), run for 71 trials, never got below 41/100 refusals (baseline 68/100). The KL divergence stayed at ~0.001, meaning the ablation barely changed anything.
• In a direct test on 48 harmful prompts (34 refused without ablation), a uniform λ = 1 on all layers gave 43 refusals, more than none at all. Uniform λ = 2 gave 0/48.

The four residual streams are mixed and re-read by every layer. Removing v from what a layer writes leaves the v-components already in the streams in place, and later layers keep carrying them forward. Only a reflection of v (λ ≈ 2, a Householder reflection that also preserves norms) removes the signal. The study behind this model therefore ran with every Heretic weight multiplied by 2, so λ ranged from 1.6 to 3.0 at the peak layer.

Method

• Tool: Heretic 2.0.0.dev0 (commit 3521f86) with a custom exllamav3 backend. Every forward pass (residuals, first-token logits, greedy generations) ran on this exact 3.05 bpw EXL3 model, with the CPU expert offload from exllamav3-rocm-moe.
• Target: thinking mode, since the chat template opens <think> by default and the model decides to refuse inside its reasoning.
• Directions: standard Heretic.
  • Per layer, the difference between the mean residual over 400 harmful prompts (mlabonne/harmful_behaviors) and over 400 harmless prompts (mlabonne/harmless_alpaca).
  • Taken at the position after a closed think block, and orthogonalized against the harmless mean.
  • The residual of a layer is the mean of its four hyper-connection streams.
  • Stored in abliteration_directions.safetensors, shape (49, 2560).
• Objectives:
  • Refusals were scored on the first 160 reasoning tokens, with reasoning-style markers ("refuse", "decline", "offer an alternative", ...).
  • KL divergence was measured on the first token over 100 harmless prompts.
• Search: 100 Optuna TPE trials, 30 of them random startup trials, seed 1337. Row normalization "none" was used for every component (see above), with the weight multiplier at 2.
• Selected trial (84): refusals 2/100, KL 0.024 in the study. An independent re-score after export gave 5/100 and KL 0.022. It ablates 36 sublayers, around the attention peak at layer 35 and the MoE peak at layer 30. The full parameters are in abliteration.json.

Evaluation

Both models were run through the same OpenAI-compatible endpoint (TabbyAPI), with thinking on, temperature 0.6, top-p 0.95, top-k 20 and one fixed seed per question. Each answer had a budget of 10K streamed tokens, and an answer that hit it was counted as wrong. Raw per-question results are in eval/.

Across all 170 paired questions the two models are statistically indistinguishable. One difference is visible: on the coding tasks the heretic model thinks longer (1,014 vs 653 tokens on average), and 4 of its answers hit the 10K budget, against 2 for the original. The refusals were 5/100 on Heretic's held-out harmful prompts. A spot check through the API (10 other harmful prompts, thinking on, first ~400 streamed chunks of each response) found a refusal phrase in 1 of 10.

Speed is unchanged. The ablation is 36 small vector operations per forward pass. A 1.4K-token code-edit request took 20.5-22.6 s with this model and 21.8 s with the original (RX 7900 XTX, TabbyAPI, MTP drafts).

Usage

The sidecar is only read by code that knows about it. Without it you get the original model: the weights are the same, and abliteration.safetensors is simply ignored.

exllamav3-rocm-moe (RX 7900 XTX / RDNA3, CPU expert offload)

phoenixhaxor/exllamav3-rocm-moe loads abliteration.safetensors natively (EXL3_ABLATION=0 turns it off). It runs this model on a single 24 GB GPU with 60+ GB of RAM: the hot experts stay in VRAM, the rest are computed on the CPU, and the n-gram table is read from the SSD. The README there has the full TabbyAPI profile:

• 192K context
• MTP drafting with prompt lookup
• vision with vision_offload
• ~85 tok/s on code edits, 50-70 tok/s in agent sessions on an RX 7900 XTX

Stock exllamav3 (any backend, no rebuild)

from exllamav3 import Config, Model, Cache, Tokenizer, Generator
import exl3_ablation                      # ships in this repo

model_dir = "Qwen3.8-Flash-Next-exl3-3.05bpw-heretic"
config = Config.from_directory(model_dir)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens = 32768)
model.load()
exl3_ablation.apply(model, model_dir)     # abliterated
# exl3_ablation.remove(model)             # back to the original model

exl3_ablation.py wraps the attention and MLP modules of every transformer block. Their outputs are exactly the sublayer outputs ablated here, including the combined MoE output. The MTP draft layer is not ablated; drafts are verified by the ablated model as usual.

TabbyAPI

With stock exllamav3, add exl3_ablation.apply(self.model, model_dir) after the model is loaded in TabbyAPI's exllamav3 backend. With exllamav3-rocm-moe, nothing is needed.

Recommended sampling is the model's own generation_config.json: temperature 1.0, top-k 20, top-p 0.95. Many agents send none of these, and without a top-k / top-p the server samples the full 248K vocabulary, which degrades long reasoning. Set them as server defaults.

Files

Disclaimer

This model will comply with requests that the original model refuses. You are responsible for how you use it and for following applicable laws.

Base model: Qwen3.8-Flash-Next by the Qwen team, under the Qwen Community License 1.0 (LICENSE). EXL3 quantization by turboderp.