gpt-oss-20b-abliterated — EXL3 6.0 bpw
An ExLlamaV3 EXL3 6.0 bpw quantization of wangzhang/gpt-oss-20b-abliterated (a refusal-suppressed openai/gpt-oss-20b, 21B total / 3.6B active, 32 experts). Baked and measured by Lna-Lab (YUKI) on 2026-09-03. Sibling: gpt-oss-120b-abliterated-EXL3-6bpw.
| Size | 17 GB — does not fit one 16 GB card; use two GPUs or -mcl (experts on CPU) |
| Bits | 6.0 bpw decoder, 8 bpw head (-hb 8 -hq -cb mul1) |
| Context | 128k (native) |
| Format | harmony — use the bundled chat_template.jinja |
Why this exists
Stock ExLlamaV3 1.4.5 mis-loads bf16 HF-layout gpt-oss experts (gate/up interleave ignored, expert biases dropped) and the quantized model speaks noise. This repo was baked with the fix described in the 120B card and verified (layer-0 MoE vs transformers reference: relative error 4e-4). OpenAI's MXFP4 checkpoints are unaffected.
Measured speed (RTX PRO 2000 Blackwell 16 GB, ExLlamaV3 1.4.5, 128 tokens, no speculation)
Single stream:
| Form | ja | en | code | VRAM |
|---|---|---|---|---|
| TP=2 (2 × 16 GB) | 63 | 75.2 | — tok/s | ~8.5 GB / GPU |
| layer-split, 2 GPUs | 50.4 | 50.2 | 50.1 tok/s | 15.2 + 2.2 GB |
1 GPU + experts on CPU (-mcl 24) |
41.4 | 44.8 | 42.0 tok/s | 2.4 GB |
Multi-stream (en, aggregate / per stream): TP=2 — 1: 75 / 75, 2: 107 / 53, 4: 121 / 30, 8: 139 / 17; layer-split — 1: 49, 2: 72, 4: 83, 8: 89. Monotonic (unlike the 120B on TP=7).
Prefill (cold, tokens/s at 256 / 1k / 4k / 16k): 2 GPUs 1,169 / 1,844 / 3,249 / 3,541 (16k in 4.6 s); 1 GPU + CPU experts 562 / 1,257 / 2,512 / 2,861 (16k in 5.7 s).
Running it
pip install exllamav3==1.4.5 # plus the SlidingAttention patch described in the 120B card
CUDA_VISIBLE_DEVICES=0,1 python3 lna-lab/serve-gptoss.py -m . -tp --port 8020 # 2 GPUs, tensor parallel
CUDA_VISIBLE_DEVICES=0 python3 lna-lab/serve-gptoss.py -m . -mcl 24 --port 8020 # 1 GPU + CPU experts
lna-lab/serve-gptoss.py is a small OpenAI-compatible harmony server (analysis → reasoning_content,
commentary tool calls → tool_calls, final → content, SSE with usage, reasoning_effort, sampling
knobs, structured tool-failure results). See the 120B card for the ExLlamaV3 1.4.5 patches and the
jinja2 tojson trap the server already works around.
Files
model-*.safetensors,config.json,tokenizer.json,chat_template.jinja,generation_config.jsonLNA-LAB-NOTES.md— recipe, traps and measurements (Japanese)lna-lab/— server, meters, bake script, upstream model card
Credits
- Base model and abliteration: wangzhang (abliterix)
- Original weights: OpenAI, Apache-2.0
- ExLlamaV3: turboderp
- Bake, measurements, server: Lna-Lab / YUKI (@Tono_Ken3)