Huihui-gemma-4-31B-it-abliterated-v2-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2 — an abliterated (uncensored) Gemma 4 31B dense model with multimodal capability.

59 GB → 20.5 GB. Single NVIDIA Blackwell GPU.

Why This Model

Gemma 4 31B Dense uses all 31B parameters on every token — unlike the 26B-A4B MoE variant (3.8B active). This means:

• Deeper reasoning per token at the cost of speed
• No routing noise — deterministic compute path
• 262K context with multimodal (vision) support
• Abliterated — no refusals for local agent workflows

Key Specs

Quickstart

Standard (vLLM)

vllm serve Lna-Lab/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90

With reasoning + tool calling

vllm serve Lna-Lab/Huihui-gemma-4-31B-it-abliterated-v2-NVFP4 \
    --max-model-len 32768 \
    --reasoning-parser gemma4 \
    --tool-call-parser gemma4 \
    --enable-auto-tool-choice

Docker

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 32768

Benchmark

Single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM).

Sustained: ~51 tok/s (single GPU, CUDA Graph PIECEWISE).

Slower than Gemma 4 26B-A4B MoE (~160 tok/s) because all 31B parameters are active per token vs. 3.8B for MoE. Trade-off: deeper reasoning per token.

Quantization Details

Recipe

from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)

Following RedHatAI/gemma-4-31B-it-NVFP4 proven recipe.

Calibration

• Dataset: neuralmagic/calibration (LLM split)
• Samples: 32
• Max sequence length: 2048

Reproduction

from transformers import Gemma4ForConditionalGeneration, AutoProcessor
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL_ID = "huihui-ai/Huihui-gemma-4-31B-it-abliterated-v2"
model = Gemma4ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)

recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:32]")
def preprocess(example):
    messages = [{"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
                for m in example["messages"]]
    return processor.apply_chat_template(messages, return_tensors="pt", padding=False,
        truncation=True, max_length=2048, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False)
ds = ds.map(preprocess, batched=False, remove_columns=ds.column_names)

def data_collator(batch):
    assert len(batch) == 1
    return {k: (torch.tensor(v) if k != "pixel_values"
                else torch.tensor(v, dtype=torch.bfloat16).squeeze(0))
            for k, v in batch[0].items()}

oneshot(model=model, recipe=recipe, dataset=ds,
        max_seq_length=2048, num_calibration_samples=32, data_collator=data_collator)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Environment

Requirements

• GPU: NVIDIA Blackwell (SM 120)
• VRAM: 21 GB minimum (model only), ~92 GB for 128K context
• Software: vLLM nightly (cu130)

Notes

• Abliterated (uncensored). Use responsibly.
• Vision tower preserved in BF16 — multimodal works.
• NVFP4 is Blackwell-specific. Will not work on Ampere/Hopper.
• Dense 31B is slower than MoE 26B-A4B (~51 vs ~160 tok/s) but reasons deeper per token.

Credits

• Base model: huihui-ai (abliteration)
• Original model: Google DeepMind (Gemma 4)
• Quantization recipe: RedHatAI (proven path)
• Quantization tool: llm-compressor

Support the Base Model Author

• Ko-fi: https://ko-fi.com/huihuiai
• Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge