Qwen3.8-27B-Uncensored — EXL3 4.0 bpw
EXL3 quantization of orcarouter/Qwen3.8-27B-Uncensored, an abliterated (refusal-removed) build of Qwen/Qwen3.8-27B.
Built with ExLlamaV3 1.4.6.
16 GB on disk — fits a single 24 GB card with room for context.
The MTP head is preserved at 8 bpw and works. GGUF conversions of this
family drop the mtp.* tensors, so they have no self-drafter. TabbyAPI logs
Using main model MTP component for drafting on load.
Sibling repos
| Variant | Size | Repo |
|---|---|---|
| 4.0 bpw | 16 GB | this one |
| 6.0 bpw | 22 GB | Qwen3.8-27B-Uncensored-exl3-6bpw |
| 8.0 bpw | 28 GB | Qwen3.8-27B-Uncensored-exl3-8bpw |
Build
| Language model | 4.00 bpw |
lm_head |
6 bpw |
| MTP layers | 8 bpw |
| Vision tower | unquantized (16-bit) |
| Embeddings | unquantized (16-bit) |
python convert.py \
-i Qwen3.8-27B-Uncensored \
-o Qwen3.8-27B-Uncensored-exl3-4bpw \
-w /tmp/exl3-work \
-b 4.0 -hb 6 -mb 8 -vb 16 \
-d 0,1,2,3 -v
Default bundled calibration corpus (wiki 50, C4 20, code 20, random tokens 20, technical 10, multilingual 10, tiny 5), 250 rows x 2048 columns.
Usage — TabbyAPI
model:
model_dir: /path/to/models
model_name: Qwen3.8-27B-Uncensored-exl3-4bpw
cache_size: 32768
cache_mode: "8,8"
tensor_parallel: true
draft_model:
draft_mode: mtp
Benchmarks
Hardware: 4x RTX 4000 Ada (20 GiB, sm89), PCIe, no NVLink, 2x Xeon
Gold 5318Y. AIPerf, streaming, ignore_eos, seed 42, with warmup.
ExLlamaV3 1.4.6 + TabbyAPI, tensor-parallel, cache_mode "8,8", MTP on.
Bitrate does almost nothing here
ISL 1000 / OSL 500 / concurrency 4 / 20 requests:
| 4.0 bpw | 6.0 bpw | 8.0 bpw | |
|---|---|---|---|
| Size | 16 GB | 22 GB | 28 GB |
| Request latency | 19,906 ms | 20,816 ms | 21,145 ms |
| ITL | 33.5 ms | 35.0 ms | 36.2 ms |
| TTFT | 3,352 ms | 3,515 ms | 3,289 ms |
| Output throughput | 97 tok/s | 91 tok/s | 90 tok/s |
75% more weight from 4 to 8 bpw costs only 6% latency. On four tensor-parallel cards over PCIe, weight bandwidth is not the bottleneck — inter-GPU synchronization dominates. Pick the bitrate on VRAM budget and quality, not on speed.
The 8 bpw TTFT being lowest of the three is noise; 20 requests can't resolve 5% gaps.
vs vLLM (same weights, FP8 build)
| vLLM 0.28.0 FP8 | EXL3 6.0 bpw | |
|---|---|---|
| Request latency | 14,158 ms | 20,816 ms |
| ITL | 26.3 ms | 35.0 ms |
| TTFT | 1,028 ms | 3,515 ms |
Long prompts (ISL 8000 / conc. 2): vLLM TTFT 5,508 ms vs 9,481 ms, prefill 1,733 vs 1,277 tok/s.
vLLM is faster in every served scenario on this box. EXL3 wins on VRAM and
on single-stream short prompts (63 vs ~45 tok/s in chat.py).
tensor_parallel is a trade, not a win
| TP off | TP on | |
|---|---|---|
| Prefill TTFT (8k) | 17,644 ms | 9,577 ms |
| Single-stream generation | 63 tok/s | 34.6 tok/s |
Helps compute-bound prefill, hurts memory-bound decode. State which one you ran when quoting EXL3 numbers.
Caveat: PCIe, no NVLink, SymmMemCommunicator unavailable on sm89. TP
results are likely better on NVLink systems. Treat these as a floor.
Quality
Not benchmarked at 4.0 bpw. For reference on the same abliterated weights, the FP8 build scores 88.0% on MMLU-Pro (business subset, 100 questions, 2 unparsed) vs 89.0% for official Qwen3.8-27B-FP8 — within noise. How 4.0 bpw compares has not been verified. At this bitrate expect measurable degradation vs the 6 and 8 bpw siblings.
Safety
Inherited from the base model: safety alignment has been substantially removed via abliteration. It will comply with requests the original Qwen3.8-27B refuses, and has no meaningful built-in guardrails. Released for research and controlled experimentation. Add your own moderation layer before any deployment. You are responsible for what you do with it.
License
Apache 2.0, inherited from Qwen/Qwen3.8-27B.
Credits
- Qwen — base model
- orcarouter — abliterated build
- turboderp — ExLlamaV3 / EXL3
- theroyallab — TabbyAPI