中文Sign up

sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

🤗 Hugging Face sourceimage-text-to-textapache-2.016.7B params20 GBsafetensors✓ 3 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 ./model-folder
Needs a seeder →

Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4

NVFP4 (W4A4) quantization of huihui-ai/Huihui-ThinkingCap-Qwen3.6-27B-abliterated — Huihui's abliterated (refusal-removed / uncensored) finetune of bottlecapai/ThinkingCap-Qwen3.6-27B, itself a token-efficient reasoning fine-tune of Qwen3.6-27B. Produced with llm-compressor → compressed-tensors, with the native MTP speculative-decode head preserved (bf16) and the Qwen3-VL vision tower preserved (bf16).

Why this pairing is nice. You keep ThinkingCap's short-<think> token efficiency (the base cuts reasoning length by ~46 % vs Qwen3.6-27B) and Huihui's abliteration (refusal directions removed), then NVFP4 + the MTP draft cut the cost of every token. Fewer thinking tokens × faster tokens × no refusal detours = a snappy, compliant local reasoner. Abliteration can shift behavior on some prompts — evaluate for your use case.

20.6 GB on disk (down from ~55.6 GB bf16). Serves on stock vLLM 0.21+ — no --quantization flag needed (auto-detected).

Architecture

Qwen3_5ForConditionalGeneration (model_type qwen3_5), dense 27.4 B:

  • Hybrid attention — Gated-DeltaNet (linear) + full-attention layers, hidden 5120, 262 K native context.
  • Vision — Qwen3-VL ViT, kept bf16; serve text-only with --limit-mm-per-prompt.
  • Native MTP (mtp_num_hidden_layers=1), kept bf16 → drives vLLM speculative decoding.
  • Thinking-by-default reasoning model (<think>…</think>, use --reasoning-parser qwen3) — but a token-efficient, abliterated one.

Quantization recipe

QuantizationModifier(targets="Linear", scheme="NVFP4",  # W4A4, group_size 16
  ignore=["lm_head", "re:.*visual.*", "re:.*conv1d.*", "re:.*mtp.*"])
  • Vision tower, DeltaNet causal conv1d, lm_head, and the entire MTP head stay bf16; everything else is NVFP4 W4A4. 32 calibration samples (neuralmagic/calibration), seq 8192, pure-CPU load (sequential-pipeline onload).
  • This model ships the MTP head as a separate model-base-aux.safetensors (bf16 tensors). Those are grafted into the NVFP4 output (model-mtp-bf16.safetensors) and spliced into the safetensors index.
  • Note for re-bakers: the grafted MTP modules must also be added to quantization_config.ignore, otherwise vLLM matches mtp.*_proj against targets=["Linear"], expects NVFP4 scales that do not exist, and loads the Qwen3_5MTP draft as garbage → 0 % spec-decode acceptance. This bake adds them automatically.

Serving (vLLM ≥ 0.21)

vllm serve sakamakismile/Huihui-ThinkingCap-Qwen3.6-27B-abliterated-NVFP4 \
  --tensor-parallel-size 4 --max-model-len 131072 \
  --max-num-seqs 16 --gpu-memory-utilization 0.90 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 --limit-mm-per-prompt '{"image":0,"video":0}' \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

On NVLink-less boxes add NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce (and NCCL_CUMEM_ENABLE=0 if TP=8 CUDA-graph capture hangs). Drop --speculative-config for plain decode. The hybrid model's KV is light (only the full-attention layers cache), so full 128 K context fits even at TP=2.

  • Reasoning model → set max_tokens ≥ 4096 (prefer 8192+). Even though ThinkingCap thinks less, at a tiny budget it can still spend it all inside <think> and return empty content.
  • Do not produce a W4A16 / NVFP4A16 variant — it fails to serve on vLLM (gptq_marlin_repack: size_n not divisible by tile_n_size=64; the odd attention-head / DeltaNet dims violate Marlin's tile constraint). W4A4 avoids Marlin (NVFP4 cutlass/FlashInfer path).
  • Sampling: the base recommends temperature=1.0, top_p=0.95, top_k=20.

License & attribution

Apache-2.0, inherited from the base models. Abliteration by huihui-ai; token-efficiency fine-tune by BottleCap AI; base Qwen3.6-27B by the Qwen Team. NVFP4 quantization by sakamakismile (Lna-Lab), reusing the validated qwen3_5 dense+MTP recipe shared with sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4 and sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4.

Support the Base Model Author (huihui-ai)

If you find the abliterated base useful, please support huihui-ai: