gpt-oss-120b-abliterated — EXL3 6.0 bpw

An ExLlamaV3 EXL3 6.0 bpw quantization of wangzhang/gpt-oss-120b-abliterated (a refusal-suppressed openai/gpt-oss-120b, 117B total / 5.1B active, 128 experts). Baked and measured by Lna-Lab (YUKI) on 2026-09-03. Sibling: gpt-oss-20b-abliterated-EXL3-6bpw.

Why this exists

The bf16 HF-layout abliterated checkpoint cannot be quantized correctly by a stock ExLlamaV3 1.4.5: its bf16 fused-expert loader splits gate_up_proj as first-half/second-half, but the HF layout interleaves gate/up along the output dim (gate = [::2], up = [1::2]), and it drops the expert biases. The result speaks noise. This repo was baked with a small fix (see below) and the layer-0 MoE output was verified against the transformers reference (relative error 4e-4, cosine 1.0000). OpenAI's MXFP4 checkpoints do not hit this path, so turboderp's EXL3 quants are unaffected.

Measured speed (RTX PRO 2000 Blackwell 16 GB × 12, ExLlamaV3 1.4.5, single stream, 128 tokens, no speculation)

Prefill (cold, tokens/s at 256 / 1k / 4k / 16k): 1 GPU + CPU experts 303 / 528 / 1,225 / 1,542 (16k in 10.6 s); TP=7 332 / 2,106 / 3,590 / 4,323 (16k in 3.8 s).

Multi-stream on TP=7 is non-monotonic in this build (aggregate: 1 stream 73, 2 → 25, 4 → 136, 8 → 80 tok/s); serve one stream or exactly four. The 20B sibling scales monotonically, so this looks specific to 128 experts split seven ways — unresolved.

Sanity: "The capital of France is" → "Paris."; harmony analysis/final channels intact; tool calls (commentary channel) round-trip with the bundled server.

Running it

pip install exllamav3==1.4.5     # torch cu128+, Blackwell/Ada/Ampere GPU
# apply patch 1 below (needed even to *load* gpt-oss in 1.4.5), then:
CUDA_VISIBLE_DEVICES=0 python3 lna-lab/serve-gptoss.py -m . -mcl 36 -cs 131072 --port 8018   # 1 GPU + CPU experts
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 python3 lna-lab/serve-gptoss.py -m . -tp --port 8018       # tensor parallel

lna-lab/serve-gptoss.py is a small OpenAI-compatible server (stdlib + jinja2) that renders harmony through the model's own template, routes analysis → reasoning_content, commentary tool calls → tool_calls, final → content, streams SSE with usage, honours reasoning_effort and sampling knobs, and adds a few harness-era conveniences for a pre-harness model (consecutive user messages folded, tool failures returned as structured JSON so it stops retrying the identical call).

ExLlamaV3 1.4.5 patches (until upstream fixes them)

1. exllamav3/modules/sliding_attn.py — in class SlidingAttention(Module): add a class attribute qsa_indexer = None (bc_attn._qsa_module_eligible reads it on sliding layers; gpt-oss has none). Without it generation fails with AttributeError: 'SlidingAttention' object has no attribute 'qsa_indexer'.
2. exllamav3/modules/linear.py, fp16 fused-expert branch ("Special dumb loading mode") — honour finterleaved when slicing frange (slice(frange[0] // n, None, full // n)) and load {fkey}_bias per expert. Only needed to bake from bf16 HF-layout gpt-oss; LNA-LAB-NOTES.md has the exact change and the verification script.
3. jinja2's built-in tojson HTML-escapes and returns Markup, which corrupts the <|channel|> tokens it is concatenated with in the tool-result turn (the model then answers !!!!). Override it with plain json.dumps(ensure_ascii=False) — the bundled server does.

Files

• model-*.safetensors, config.json, tokenizer.json, chat_template.jinja, generation_config.json
• LNA-LAB-NOTES.md — bake recipe, patches, traps, all measurements (Japanese)
• lna-lab/ — serve-gptoss.py (server), exl3-bench.py / prefill.py (the meters), bake script, raw bench logs, and the upstream model card (README.upstream-wangzhang.md)

Credits

• Base model and abliteration: wangzhang (abliterix)
• Original weights: OpenAI, Apache-2.0
• ExLlamaV3: turboderp
• Bake, patches, measurements, server: Lna-Lab / YUKI (@Tono_Ken3)