Huihui-Qwen3.6-27B-abliterated-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-Qwen3.6-27B-abliterated — an abliterated (uncensored) variant of Qwen3.6-27B, the dense 27B VLM with Gated DeltaNet hybrid attention.

Quantized to NVIDIA FP4 by Lna-Lab using custom Blackwell NVFP4 GEMM kernels (lna-lab/blackwell-geforce-nvfp4-gemm).

55.6 GB → 19.7 GB (0.35x) — vision tower preserved in BF16. Runs on a single NVIDIA Blackwell GPU.

Key Specs

| | |

|---|---|

| Base model | huihui-ai/Huihui-Qwen3.6-27B-abliterated |

| Original | Qwen/Qwen3.6-27B |

| Architecture | Dense 27B, Gated DeltaNet + Gated Attention hybrid, VLM |

| Quantization | NVFP4 (W4A4 — weights FP4, activations FP4, scales FP8) |

| Format | compressed-tensors (native vLLM support) |

| Tool | vllm-project/llm-compressor + blackwell-geforce-nvfp4-gemm |

| Size | 19.7 GB |

| Requires | NVIDIA Blackwell GPU (SM 120), vLLM >= 0.19 |

Benchmark Results

Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB), vLLM 0.19.1+, 128K context, FP8 KV cache.

| Task | Tokens | Speed (tok/s) | Status |

|------|--------|---------------|--------|

| English reasoning | 1,024 | 56.2 | PASS |

| Japanese essay (方丈記) | 2,048 | 59.7 | PASS |

| Python code generation | 2,048 | 59.1 | PASS |

| Contradictory instructions | 1,500 | 59.5 | PASS |

| VLM image description | 947 | 58.1 | PASS |

| Math proof (√2 irrationality) | 1,024 | 59.3 | PASS |

Sustained throughput: ~58 tok/s (single GPU, 128K context, FP8 KV cache)

VRAM Usage

| State | GPU Memory |

|-------|-----------|

| After model load | 92,142 MiB |

| Peak (during inference) | 92,150 MiB |

Quick Start — From Scratch with Docker

1. Pull the model

hf download sakamakismile/Huihui-Qwen3.6-27B-abliterated-NVFP4 \
  --local-dir /models/Huihui-Qwen3.6-27B-abliterated-NVFP4

2. Run with Docker (128K context + FP8 KV cache)

docker run -d --name huihui-qwen36-27b \
  --gpus '"device=0"' --shm-size=16g \
  -v /models/Huihui-Qwen3.6-27B-abliterated-NVFP4:/models/current:ro \
  -p 8000:8000 \
  vllm/vllm-openai:cu130-nightly \
  --model /models/current \
  --trust-remote-code --quantization modelopt --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"preserve_thinking":true}' \
  --enable-prefix-caching --enable-chunked-prefill \
  --max-model-len 131072 --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8_e4m3

3. Run with vLLM directly

vllm serve /models/Huihui-Qwen3.6-27B-abliterated-NVFP4 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.95 \
  --dtype auto \
  --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code

4. Test inference

# Text
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Huihui-Qwen3.6-27B-abliterated-NVFP4",
    "messages": [{"role": "user", "content": "Write a haiku about quantization."}],
    "max_tokens": 256,
    "temperature": 0.0
  }'

# VLM (image input)
import base64, requests
from pathlib import Path

img_b64 = base64.b64encode(Path("photo.jpg").read_bytes()).decode()
resp = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "Huihui-Qwen3.6-27B-abliterated-NVFP4",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}},
        {"type": "text", "text": "Describe this image."},
    ]}],
    "max_tokens": 1024,
})
print(resp.json()["choices"][0]["message"]["content"])

Quantization Details

Recipe

QuantizationModifier:
  targets: [Linear]
  ignore: [lm_head, 're:.*visual.*', 're:.*mlp.gate$', 're:.*mlp.shared_expert_gate$']
  scheme: NVFP4

What's Quantized / What's Not

• Quantized (NVFP4): All Linear layers in the language model
• Kept in BF16: lm_head, all vision layers (model.visual.*), MLP gates

Reproduction

from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

model = Qwen3_5ForConditionalGeneration.from_pretrained(
    "huihui-ai/Huihui-Qwen3.6-27B-abliterated",
    torch_dtype=torch.bfloat16, trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained(
    "huihui-ai/Huihui-Qwen3.6-27B-abliterated", trust_remote_code=True,
)

recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4",
    ignore=["lm_head", "re:.*visual.*", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],
)

# Calibration with neuralmagic/calibration dataset (20 samples, 8192 seq len)
# ... (see quantization script in repo)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Note: After saving, verify vision tower keys use model.visual. prefix (not model.language_model.visual.). See this fix for details.

Tested Environment

| Component | Version |

|-----------|---------|

| vLLM | 0.19.1rc1+ (nightly) |

| Transformers | 5.5.4 |

| PyTorch | 2.11.0+cu130 |

| llm-compressor | 0.1.dev5 |

| CUDA | 13.0 |

| GPU | NVIDIA RTX PRO 6000 Blackwell (96 GB) |

| OS | Ubuntu 24.04, Linux 6.17 |

Credits

• Original model: Qwen Team (Alibaba Group) — Qwen3.6-27B
• Abliteration: huihui-ai — Huihui-Qwen3.6-27B-abliterated
• NVFP4 quantization & benchmarking: Lna-Lab
• Blackwell NVFP4 GEMM kernels: lna-lab/blackwell-geforce-nvfp4-gemm
• Quantization framework: vllm-project/llm-compressor

Support the Base Model Authors

If you find this model useful, please consider supporting:

• huihui-ai (abliteration): Ko-fi | BTC: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge
• Qwen Team (original model): Star the Qwen repo

License

This model inherits the Apache 2.0 license.