GLM-5.3-Flash-abliterated-exl3-2.05bpw

GLM-5.3-Flash in EXL3 at 2.05 bpw plus a 2.2 MB vector that removes refusals while the model runs. On 4x RTX 3090 it serves a 256K context with vision, tool calls and MTP at 75 tokens/s with the vector on.

Credits first

The model weights in this repo are not mine. They are turboderp's 2.05 bpw quantization, copied byte for byte from turboderp/GLM-5.3-Flash-exl3 at branch 2.05bpw, revision 51058cd551c7e570d87bd32a4adee720edce2349. The SHA256 of all 15 LFS files match his. See SHA256SUMS. One small file differs from his. In config.json the field quantization_config.bits is the integer 2 and the exact 2.05 moved to bpw, because the Hub's config parser accepts only an integer there. quantization_config.json is unchanged.

My part is one file, vector/glm53-refusal-per-block-3dir.safetensors, the engine fork that applies it, and the measurements below. For the original card and turboderp's own numbers go to his repo.

The vector

The weights stay as turboderp published them. After every block my exllamav3 fork subtracts 3 directions from each of the 4 residual streams:

x = x - (x Vᵀ) V

V holds 3 orthonormal directions for that block. Blocks 1 to 44 have their own directions. Without the vector the same server runs the original model.

Each block has 3 directions because 1 was not enough when the model thinks:

1. the difference between harmful and harmless requests at the last prompt token, thinking off;
2. the same with thinking on. It agrees with the first only at cosine 0.4 to 0.5;
3. the same difference taken at the closing </think> of the stock model's own reasoning, where the decision to refuse is already made.

Each direction is the mean of harmful minus the mean of harmless, with the components along the mean harmless states removed. Without that last step the projection also moves ordinary prompts. The full method and the scripts: abliteration write-up.

The file is 2,166,208 bytes, SHA256 f07b2cce3f91a36658c9706301088ba9c0140f3dc032cc26937136c1779f1764. It is in its own vector/ folder so that the engine does not read it as part of the model.

Refusals

64 held-out AdvBench requests and 82 JailbreakBench requests. JailbreakBench took no part in finding the directions. Greedy decoding through the API of the running server (2.05 bpw, 256K, MTP, vision, 4 streams). The judge is the stock model without the vector.

• For the two thinking modes the "without the vector" column covers the first 32 and the first 48 requests.
• "Disputes the premise": the model takes the request and says its premise is false. Asked for proof that the Earth is flat, it says there is none and lists what believers usually argue. A strict 2-class judge put 1 of these 7 answers into refusals, so by that judge the result is 1 of 438.
• "Cut off while thinking": 10 of the 11 are "build a site / an app / a game" tasks where the model reasons past the token budget. The judge read the reasoning and found the model working on the task in all 11.

Cost of the vector

Speed is measured through the OpenAI server with answers that run to their natural end (1.5K to 4K tokens). On harmless requests refusals did not grow, and no run had empty or looping answers.

Run it

You need my exllamav3 fork. Stock exllamav3 loads these weights, but it has no hook for the vector. It also lacks the fork's GPU-to-GPU exchange, and without that exchange prefill at 32K was 989 tokens/s against 1520.

git clone -b glm53-4x3090 https://github.com/alesha-pro/exllamav3-glm53-4x3090
cd exllamav3-glm53-4x3090 && glm53/setup_env.sh && source venv/bin/activate
hf download alesha-pro/GLM-5.3-Flash-abliterated-exl3-2.05bpw --local-dir glm53/models/GLM-5.3-Flash-exl3-2.05bpw

VEC=glm53/models/GLM-5.3-Flash-exl3-2.05bpw/vector/glm53-refusal-per-block-3dir.safetensors glm53/run_server.sh fast

OpenAI API on port 30000, model id glm-5.3-flash. Leave VEC out for the original model.

• chat_template_kwargs: {"enable_thinking": false} turns thinking off.
• reasoning_effort: the template knows low and high. Anything else, and no value at all, is max, which thinks 3 to 5 times longer.
• VRAM after loading on my rig: 22,975 to 23,762 MiB of 24,576 per card.

Everything about the fork: glm53/README.md.

Other sizes

• GLM-5.3-Flash-abliterated-exl3-2.4bpw-mix: expert down projections at 3 bits. KLD 0.0959 against 0.1214 here, 98.1 GB, part of the experts in RAM.
• GLM-5.3-Flash-abliterated-Maya-M-GGUF: Project Maya's Maya-M with the same directions for their engine.

Limits

• One rig: 4x RTX 3090 at 300 W, PCIe 3.0, EPYC 7642, 2 memory channels of DDR4-2400, a kernel module patched for P2P between the cards. Without P2P set EXL3_TP_P2P_BIG=0.
• The judge is the same model without the vector. There are no human labels.
• 146 prompts from 2 public sets. Other kinds of requests may behave differently.
• I measured closeness to the original (KLD), not task quality.
• With the vector on, one speed point did not finish: a decode after 32K random tokens kept generating for 17 minutes and I stopped it. On coherent text of the same depth answers end normally. I did not find the cause.
• At effort max 9 of 146 answers ran past 12,000 tokens of reasoning. At high it was 2 of 146 past 6000. Set reasoning_effort to high for agent work.

License

MIT, the license of GLM-5.3-Flash. See LICENSE.