Huihui-Qwen3.8-27B-abliterated-FP8 (v2, requantized 2026-11-02)
FP8 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated,
produced with llm-compressor (FP8_BLOCK), for vLLM.
huihui-ai publish BF16 weights (55.6 GB) and a GGUF build for this model, no FP8. This repo is the same model at 29.09 GiB, with the vision tower and the MTP drafter intact.
Requantized on 2026-11-02. If you downloaded this repo before that date, download it again. The previous build had two defects:
- Corrupted norm weights. llm-compressor 0.13 converts Qwen3.5's offset-style RMSNorm weights (
1 + w) for its pipeline and converts them back in BF16. That rounds everyinput_layernorm/post_attention_layernorm/q_norm/k_normweight to a ~0.004 grid (2^-9), with up to ~7% relative error on individual values. In the previous build 100% of those values sat on that grid. In the source weights 22.1% do, which is what you expect by chance. This build takes every non-quantized tensor byte for byte from the source, and its share is 22.1% again.- Wrong module coverage and wrong claims. The previous build left all of
linear_attn.*in BF16 (34.25 GiB, about 7% slower decode than this build), and this card said the officialQwen/Qwen3.8-27B-FP8release does the same and uses per-tensor scales. Both claims were wrong: the official release quantizesin_proj_qkv/in_proj_z/out_projof the linear-attention layers, with 128×128 block scales. This build matches it.The source weights are unchanged: upstream revision
739e3c5b. The repo history was squashed, so the previous build is no longer on any branch. If you pinned an old commit hash, move to the current one.
| Base revision | 739e3c5b89849f6c238ce1e5b70008612ae42cdd (2026-08-24 upstream re-release, ablation narrowed to layers 18-51) |
| Size | 29.09 GiB (31.24 GB), 8 shards |
Scheme
Same scheme and the same module coverage as the official Qwen/Qwen3.8-27B-FP8 release, so it runs on the same well-trodden vLLM kernel path.
| Weights | FP8 e4m3, static, 128×128 blocks |
| Activations | FP8 e4m3, dynamic, per-token groups of 128 |
| Format | compressed-tensors (float-quantized) |
| Calibration | none (data-free) |
400 Linear modules are quantized:
q_proj/k_proj/v_proj/o_projof the 16 full-attention layersin_proj_qkv/in_proj_z/out_projof the 48 gated-delta-net (linear-attention) layersgate_proj/up_proj/down_projof all 64 MLPs
Kept in BF16: in_proj_a/in_proj_b, conv1d, all norms, embed_tokens, lm_head, the
whole vision tower, and the MTP head (mtp.*, so MTP speculative decoding works).
Every non-quantized tensor is byte-identical to upstream (all 799 of them, checked tensor by tensor), and so are the tokenizer, preprocessor configs, generation config and chat template. The chat template is in turn byte-identical to the stock Qwen3.8-27B one.
Serving with vLLM
vllm serve batsclamp/Huihui-Qwen3.8-27B-abliterated-FP8-v2 \
--max-model-len 262144 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20}'
The shipped template uses Qwen's XML tool-call format, so the parser must be qwen3_coder
(not hermes). Sampling defaults follow the upstream thinking-mode recommendation.
The chat template accepts only reasoning_effort values xhigh, medium and low, and
raises on anything else (high, max, minimal), which an OpenAI-compatible server turns
into HTTP 400. If your client sends other values, map them in a small Jinja prelude before
the template.
Measured on one DGX Spark (GB10)
vLLM 0.32.0rc1 (nightly of 2026-10-16), --max-model-len 262144, MTP k=3,
--gpu-memory-utilization 0.55, kv-cache-dtype auto, no other GPU process resident:
| Weights resident | 29.5 GiB |
| KV cache | 28.4 GiB, ~412k tokens (1.57× concurrency at 262k) |
| Decode, single stream, no thinking | 14.3 tok/s |
| TTFT, short prompt | 189 ms |
| MTP mean acceptance length | 2.5-3.9 |
| Kernel | CutlassFp8BlockScaledMMKernel |
Measured earlier on the same box and image: the previous build decoded at 13.4 tok/s, and
stock Qwen/Qwen3.8-27B-FP8 at 15.0 tok/s. Being 5 GiB heavier, the previous build also
left 5 GiB less for the KV cache at the same --gpu-memory-utilization.
Smoke-tested on vLLM: every reasoning_effort level and enable_thinking on/off return a
non-empty answer, tool calls (single, parallel, multi-turn with a tool result, and Python
code inside a streamed tool argument, 15/15) parse cleanly, JSON-schema structured output
and image input work.
Caveats
- This is an abliterated model: refusal behaviour has been removed upstream. Safety properties are not those of the original Qwen release.
- Quantization was verified structurally (tensor by tensor against the source) and by generation and tool-calling tests on vLLM; no perplexity or benchmark-suite numbers are claimed.
- With
reasoning_effort: xhighthis model family can spend a very large output budget inside the reasoning block. If you capmax_tokensbelow what it needs you getfinish_reason: lengthand an emptycontent. This is not specific to this quantization or to abliteration; stockQwen/Qwen3.8-27B-FP8behaves the same way.