Huihui gemma-4-26B-A4B-it abliterated · NVFP4 (vLLM-ready, 768)

An uncensored and full-strength build of gemma-4-26B-A4B in NVFP4 (W4A4) that loads on a stock vLLM with no source patches.

Two things make this build worth picking:

1. It serves on stock vLLM. The existing 704-intermediate NVFP4 quant fails to load on a stock vLLM with:

NotImplementedError: ('Intermediate size padding for w1 and w3, for %s NvFp4 backend,
but this is not currently supported', 'VLLM_CUTLASS')

This repo bakes the kernel alignment in: the MoE intermediate is zero-padded 704 → 768 offline, on the packed FP4 weights — a mathematically loss-less transform that inherits the original (already-correct) activation scales and needs no re-quant, no GPU, no calibration. 768 is a multiple of 128, so the CUTLASS NVFP4 MoE kernel accepts it directly (and the TP>1 / PP>1 gated-MoE NotImplementedError is sidestepped).

2. Abliterated without the usual quality tax. A common worry with uncensored models is that the modification hurts capability. We measured it. On HumanEval+ (EvalPlus strict tests, 163 problems) this lightly abliterated build scores the same as the unmodified official model — far above a heavily-modified variant:

You keep the uncensored behavior and the full coding ability.

Specs

• Format: NVFP4 (FP4 weights and activations, compressed-tensors).
• Arch: gemma4 MoE — 128 experts, top-8, hidden 2816, moe_intermediate 768, 30 layers, sliding-window attention. Context up to 256K.
• VRAM: weights are ~15.3 GiB — they do not fit a single 16 GB card (no room left for KV → OOM). Run on 2× 16 GB (--tensor-parallel-size 2) or a single ≥ 20 GB GPU.

Throughput (measured, RTX PRO 2000 Blackwell 16 GB, CUDA graph, fp8 KV)

TP=2 (2× 16 GB): single-user ≈ 99 tok/s; energy-efficient.

TP=4 (4× 16 GB): faster single-stream + higher peak aggregate.

(Aggregate = summed over concurrent requests. TP=2 wins tok/joule — half the GPUs, half the power; TP=4 wins single-stream latency and peak throughput. Power stays ~flat across concurrency: batching is the efficiency win.)

Quick start — from zero to inference

NVIDIA Blackwell GPU(s) + Docker with the NVIDIA Container Toolkit. NVFP4 is auto-detected; no --quantization flag.

2× 16 GB (tensor parallel, no NVLink) — the typical setup:

docker run --gpus all -p 8000:8000 -v $(pwd):/model:ro \
  -e NCCL_P2P_DISABLE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  vllm/vllm-openai:cu130-nightly \
  /model --served-model-name gemma \
  --tensor-parallel-size 2 --disable-custom-all-reduce \
  --gpu-memory-utilization 0.85 --max-num-seqs 32 --max-num-batched-tokens 4096 \
  --max-model-len 8192 --kv-cache-dtype fp8 \
  --limit-mm-per-prompt '{"image":0,"video":0,"audio":0}'

For TP=4 set --tensor-parallel-size 4 and --gpu-memory-utilization 0.90. For a single ≥24 GB GPU drop the --tensor-parallel-size/NCCL lines.

Inference flags that matter:

• Keep CUDA graph ON (do not pass --enforce-eager) — --gpu-memory-utilization 0.85 + --max-num-seqs 32 leave room for the ~0.3 GiB capture and give ~10× faster decode than eager.
• NCCL_P2P_DISABLE=1 + --disable-custom-all-reduce are required for tensor-parallel on PCIe/NODE topology (no NVLink).
• --kv-cache-dtype fp8 for long-context capacity (~11 full-128K requests fit on 4×16 GB).
• --limit-mm-per-prompt '{"image":0,"video":0,"audio":0}' to serve text-only and skip multimodal profiling.

Talk to it (instruct model — use the chat endpoint):

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"gemma","messages":[{"role":"user","content":"侘び寂びを一文で。"}],
  "max_tokens":256,"temperature":0.7}'

Use /v1/chat/completions (the Gemma chat template is applied). Raw /v1/completions on an instruct model yields degenerate output.

How it was made

Offline FP4 surgery on the 704 NVFP4 checkpoint: per expert, {gate,up}_proj weight+scale are padded 704→768 on the output dim and down_proj on the input dim with FP4/FP8 0x00 (=+0.0); every *_global_scale and all non-expert tensors are copied verbatim. Loss-less (no MoE-intermediate norm; gelu(0)·0=0; padded down_proj columns multiply zero weights). Verified: input_global_scale median ≈ 322, zero 1.0/0.0 sentinels, expert shapes 768-aligned.

This is an abliterated (uncensored) model — safety training removed. Use responsibly.

Credits

• Base model (abliteration): huihui-ai
• Original model: Google DeepMind (Gemma 4)
• NVFP4 quantization & benchmarking: Lna-Lab
• 704→768 kernel-alignment padding: Lna-Lab (this repo)
• Quantization method follows RedHatAI/gemma-4-26B-A4B-it-NVFP4 (proven path)
• Quantization framework: vllm-project/llm-compressor

Support the Base Model Author

If you find this model useful, please consider supporting huihui-ai — the creator of the abliterated base model:

• Ko-fi: https://ko-fi.com/huihuiai
• Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge