Sign up

sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

🤗 On Hugging Faceimage-text-to-textapache-2.016.7B params20 GBsafetensors✓ Checksum-verifiedupdated 0d ago
Magnet

Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

NVFP4 (W4A4) quantization of huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated — Huihui's abliterated (refusal-removed / uncensored) finetune of bottlecapai/ThinkingCap-Qwen3.6-27B, itself a token-efficient reasoning fine-tune of Qwen3.6-27B. Produced with llm-compressor → compressed-tensors, with the native MTP speculative-decode head preserved (bf16) and the Qwen3-VL vision tower preserved (bf16).

Why this pairing is nice. You keep ThinkingCap's short-`` token efficiency (the base cuts reasoning length by ~46 % vs Qwen3.6-27B) and Huihui's abliteration (refusal directions removed), then NVFP4 + the MTP draft cut the cost of every token. Fewer thinking tokens × faster tokens × no refusal detours = a snappy, compliant local reasoner. Abliteration can shift behavior on some prompts — evaluate for your use case.

20.6 GB on disk (down from ~55.6 GB bf16). Serves on stock vLLM 0.21+ — no --quantization flag needed (auto-detected).

Architecture

Qwen3_5ForConditionalGeneration (model_type qwen3_5), dense 27.4 B:

  • Hybrid attention — Gated-DeltaNet (linear) + full-attention layers, hidden 5120, 262 K native context.
  • Vision — Qwen3-VL ViT, kept bf16; serve text-only with --limit-mm-per-prompt.
  • Native MTP (mtp_num_hidden_layers=1), kept bf16 → drives vLLM speculative decoding.
  • Thinking-by-default reasoning model (…, use --reasoning-parser qwen3) — but a token-efficient, abliterated one.

Quantization recipe

QuantizationModifier(targets="Linear", scheme="NVFP4",  # W4A4, group_size 16
  ignore=["lm_head", "re:.*visual.*", "re:.*conv1d.*", "re:.*mtp.*"])
  • Vision tower, DeltaNet causal conv1d, lm_head, and the entire MTP head stay bf16; everything else is NVFP4 W4A4. 32 calibration samples (neuralmagic/calibration), seq 8192, pure-CPU load (sequential-pipeline onload).
  • This model ships the MTP head as a separate model-base-aux.safetensors (bf16 tensors). Those are grafted into the NVFP4 output (model-mtp-bf16.safetensors) and spliced into the safetensors index.
  • Note for re-bakers: the grafted MTP modules must also be added to quantization_config.ignore, otherwise vLLM matches mtp.*_proj against targets=["Linear"], expects NVFP4 scales that do not exist, and loads the Qwen3_5MTP draft as garbage → 0 % spec-decode acceptance. This bake adds them automatically.

Serving (vLLM ≥ 0.21)

vllm serve sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \
  --tensor-parallel-size 4 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

On NVLink-less boxes add NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce (and NCCL_CUMEM_ENABLE=0 if TP=8 CUDA-graph capture hangs). Drop --speculative-config for plain decode. The hybrid model's KV is light (only the full-attention layers cache), so full 128 K context fits even at TP=2.

  • Reasoning model → set max_tokens ≥ 4096 (prefer 8192+). Even though ThinkingCap thinks less, at a tiny budget it can still spend it all inside `` and return empty content.
  • Do not produce a W4A16 / NVFP4A16 variant — it fails to serve on vLLM (gptq_marlin_repack: size_n not divisible by tile_n_size=64; the odd attention-head / DeltaNet dims violate Marlin's tile constraint). W4A4 avoids Marlin (NVFP4 cutlass/FlashInfer path).
  • Sampling: the base recommends temperature=1.0, top_p=0.95, top_k=20.

License & attribution

Apache-2.0, inherited from the base models. Abliteration by huihui-ai; token-efficiency fine-tune by BottleCap AI; base Qwen3.6-27B by the Qwen Team. NVFP4 quantization by sakamakismile (Lna-Lab), reusing the validated qwen3_5 dense+MTP recipe shared with sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4 and sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4.

Support the Base Model Author (huihui-ai)

If you find the abliterated base useful, please support huihui-ai: