Huihui-Qwen3.8-27B-abliterated — AutoRound INT4, baked head/MTP, INT8 embedding

A 4-bit build of huihui-ai/Huihui-Qwen3.8-27B-abliterated (an abliterated, i.e. refusal-removed, version of Qwen/Qwen3.8-27B) made for a single Intel Arc Pro B70 (32 GB) under vLLM XPU. It follows the recipe of ukisai/Swift-1.5-Qwen3.8-27b-W4A16-AutoRound for the body and of Wrapzii's Swift 1.5 GPTQ-Int4-baked-v1-embed-int8 for the head, MTP and embedding — applied to the uncensored Huihui weights.

Uncensored model. Abliteration removes most refusal behaviour. It will follow requests the original model refuses. You are responsible for how you use it.

What is in the files

Weights resident on the GPU: 14.75 GiB (vs 18.28 GiB for a GPTQ build that keeps lm_head / MTP dense), which leaves room for the full 262,144-token context on one 32 GB card (vLLM reports a 319K-token KV pool with fp8 KV and 400K with the K8/V4 int8-K / int4-V KV cache).

The dense embedding is still in the shards, so stock loaders work; the INT8 embedding only takes effect in a runtime that reads embed_tokens_quant (the patched image below).

Quality

WikiText-2 test perplexity, 40 chunks x 2,048 tokens (81,880 scored tokens), the same token chunks for both, computed from vLLM prompt_logprobs on the B70 (fp8 KV; with the K8/V4 KV cache it measures 6.276):

Head / MTP bake on held-out activations (relative output L2 error; GPTQ vs round-to-nearest):

These are matrix and perplexity checks, not a full benchmark suite; upstream evaluation scores have not been re-established for this derivative. A quick functional check (arithmetic, thinking on/off, tool-call template, an uncensored prompt) passed.

Speed on one Arc Pro B70

Numbers are BetterBench median decode tokens/s (temperature 0.7, top-p 0.95, thinking on) and prefill tokens/s, all on one 32 GB card.

Recommended: K8/V4 KV cache + the built-in MTP head (3 speculative tokens), 262,144-token context. The KV cache is int8-K / int4-V (k8v4-xpu, ported to one GPU in davetha/k8v4-xpu tp1-k8v4); every attention layer, including MTP, uses the same cache. The KV pool holds 400,705 tokens (1.5 full-length requests); WikiText-2 perplexity 6.276 (6.266 with fp8 KV).

The K8/V4 + MTP3 column is a short run (5 passes per category, the others 20), so differences under ~5% are noise.

Time to first token with K8/V4 + MTP3: 0.8 s at 2K, 6.3 s at 16K, 30 s at 64K. Greedy single runs: prose ~73, JSON ~99, code ~93 tok/s; 72K-token needle retrieval passes.

Why not the DFlash2 drafter? With the z-lab DFlash2 drafter decode is 7-15% faster, but at 256K it only fits next to K8/V4 by running the drafter's own layers on fp8 (a vLLM patch, vllm-patches/), i.e. two KV formats sharing one block pool. That setup produced one degenerate repetition loop in real use (a single token repeated from the first reasoning token; draft acceptance dropped to zero) that could not be reproduced in ~700 test requests, so it is not recommended. fp8 KV + DFlash2 has no such mixing and was fine, but tops out near 192K context.

Serving (vLLM XPU, one Arc Pro B70)

Build the image from davetha/k8v4-xpu tp1-k8v4 (TP1-B70.md: Dockerfile.tp1, then k8v4_v030/Dockerfile.tp1-kv; the vLLM skip patch is not needed). The single-library version of the same port is proposed upstream in Wrapzii/k8v4-xpu#1.

docker run -d --name qwen27-b70 \
  --device /dev/dri/card1 --device /dev/dri/renderD128 --group-add render \
  --ipc=host --shm-size=8g -p 8200:8000 \
  -v /path/to/this/repo:/model:ro \
  -e VLLM_TARGET_DEVICE=xpu -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \
  -e XE2_KV_S2_NSG_DRAFT=32 -e XE2_KV_S2_NSG_VERIFY=8 -e K8V4_PREFILL_GATHER=triton -e K8V4_SDPA_ASYNC=0 \
  vllm-openai-xpu:v0.30.0-k8v4-tp1-kv /model \
  --host 0.0.0.0 --port 8000 --served-model-name qwen27-huihui-unc \
  --gpu-memory-utilization 0.92 --dtype bfloat16 \
  --max-model-len 262144 --kv-cache-dtype int8_k_int4_v --tensor-parallel-size 1 \
  --max-num-seqs 4 --max-num-batched-tokens 4224 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --enable-prefix-caching --trust-remote-code \
  --limit-mm-per-prompt '{"image":4,"video":0}' --mm-processor-kwargs '{"max_pixels":1048576}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,12,16]}'

For fp8 KV instead, use the vllm-openai-xpu:v0.30.0-k8v4-tp1 image with --kv-cache-dtype fp8 (319K-token pool). A stock vLLM XPU image loads the checkpoint too, with the embedding dense.

Thinking: chat_template_kwargs: {"enable_thinking": false} turns it off; reasoning_effort takes low / medium / xhigh (the bundled chat_template_low.jinja defaults to low). Tool calls use the qwen3 XML format.

How it was made

1. Body — AutoRound 0.16.0 on the bf16 checkpoint, on the B70 (2.5 h, 27.5 GB peak VRAM):

   auto-round --model huihui-ai/Huihui-Qwen3.8-27B-abliterated --scheme W4A16 --bits 4 --group_size 128 \
     --iters 200 --batch_size 8 --nsamples 256 --seqlen 2048 --seed 42 \
     --dataset HuggingFaceH4/ultrachat_200k \
     --ignore_layers "lm_head,.*visual.*,.*mtp.*,.*in_proj_a,.*in_proj_b" \
     --format auto_round --low_gpu_mem_usage
   

   AutoRound writes the 15 MTP tensors (dropped by the Transformers loader) back from the source into model_extra_tensors.safetensors.
2. GPTQ compatibility — tools/prepare_swift_autoround.py from k8v4-xpu: verifies the 400 packed linears and zero points, adds deterministic g_idx, switches the config to gptq. No weight values change.
3. Calibration — tools/calibrate_swift_bake.py (run at TP1 with fp8 KV on the single card): 128 WikiText-2 validation prompts of 384 tokens, 64 generated tokens each, MTP on; Hessians for lm_head and the MTP inputs, every 64th row held out.
4. Bake — tools/bake_gptq_resume.py: symmetric GPTQ INT4, group 128, 1% damping, no activation ordering.
5. Assemble — tools/assemble_swift_bake.py: rewritten shards without the dense replaced tensors, the 36 GPTQ arrays in model-bake-int4.safetensors, the INT8 embedding side file. Settings and per-linear errors are in BAKE.json.

Credits and licenses

• Base model: Qwen/Qwen3.8-27B (Apache-2.0); abliteration: huihui-ai (Apache-2.0, LICENSE).
• Body quantization recipe: ukisai/Swift-1.5-Qwen3.8-27b-W4A16-AutoRound; tool: Intel AutoRound.
• Head/MTP bake, INT8 embedding and XPU serving stack: Wrapzii/k8v4-xpu; its GPTQ bake tooling is adapted from launch80/B65 (MIT, B65-LICENSE).
• This derivative: weights requantized only — no fine-tuning, no merging.