Sign up

sakamakismile/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4

🤗 On Hugging Facetext-generationapache-2.015.1B params16 GBsafetensors✓ Checksum-verifiedupdated 0d ago
Magnet

Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated — an abliterated (uncensored) variant of Google's Gemma 4 26B-A4B Mixture-of-Experts model.

49 GB → 16.5 GB — runs on a single NVIDIA Blackwell GPU at ~160 tok/s.

### Known Issue: Japanese / Non-English Long-Form Generation
NVFP4 quantization of the 26B (128-expert) model causes intermittent repetition collapse on long Japanese text generation. The model may produce degenerate output like get(get) get(get)... when generating 500+ tokens in Japanese. English tasks (code, math, reasoning) are unaffected.
This appears to be inherent to the 128-expert MoE architecture under FP4 quantization — with fewer experts, quantization noise can corrupt the specific expert combinations needed for non-English generation.
For multilingual / Japanese workloads, we recommend the 48B (256-expert) variant instead: sakamakismile/Huihui4-48B-A4B-abliterated-NVFP4 — same inference speed, stable across all languages.

Key Specs

| | |

|---|---|

| Base model | huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated |

| Architecture | Gemma 4 MoE — 25.2B total, 3.8B active per token |

| Quantization | NVFP4 (W4A4 — weights FP4, activations FP4, scales FP8) |

| Format | compressed-tensors (native vLLM support) |

| Tool | vllm-project/llm-compressor (main) |

| Size | 16.5 GB (single safetensors shard) |

| Requires | NVIDIA Blackwell GPU (SM 120), vLLM nightly (cu130) |

Quickstart

vLLM (recommended)

vllm serve Lna-Lab/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4 \
    --max-model-len 8192

No --quantization flag needed — vLLM auto-detects compressed-tensors format.

Docker

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 8192

Python (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(
    model="Lna-Lab/Huihui-gemma-4-26B-A4B-it-abliterated-NVFP4",
    max_model_len=8192,
    gpu_memory_utilization=0.85,
)

output = llm.generate(
    ["Explain quantum entanglement in simple terms."],
    SamplingParams(max_tokens=256, temperature=0.7),
)
print(output[0].outputs[0].text)

Benchmark

Tested on a single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM), vLLM 0.19.1+, max_model_len=8192, temperature=0.0.

| Task | Tokens | Time (s) | Speed (tok/s) | Quality |

|------|--------|----------|---------------|---------|

| Japanese essay (方丈記) | 2,048 | 14.05 | 145.8 | FAIL (repetition collapse) |

| Python code generation | 1,904 | 12.04 | 158.1 | PASS |

| Math reasoning (EN) | 1,090 | 6.86 | 158.8 | PASS |

| VLM image description | 271 | 1.80 | 150.2 | PASS |

Sustained throughput: ~150–160 tok/s (post-warmup, single GPU).

VRAM Usage

| State | GPU Memory |

|-------|-----------|

| After model load | 89,678 MiB |

| Peak (during inference) | 90,074 MiB |

Comparison with 48B (256-expert) variant

| | 26B (this model) | 48B-NVFP4 |

|---|---|---|

| Experts | 128 | 256 |

| Active params | 3.8B | 3.8B (same) |

| Speed | ~158 tok/s | ~153 tok/s |

| Disk size | 16.5 GB | 27.3 GB |

| English quality | Good | Good |

| Japanese quality | Unstable | Stable |

| VLM | Good | Good |

Quantization Details

Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head, 're:.*embed.*', 're:.*router', 're:.*vision_tower.*']
      scheme: NVFP4

What's quantized, what's not

  • Quantized (NVFP4): All Linear layers in the language model, including MoE expert layers
  • Kept in BF16: lm_head, all embedding layers, MoE routers, entire vision tower

Calibration

  • Dataset: neuralmagic/calibration (LLM split)
  • Samples: 20
  • Max sequence length: 8192
  • MoE expert calibration handled automatically by llm-compressor's SequentialGemma4TextExperts

Reproduction

from datasets import load_dataset
from transformers import AutoProcessor, Gemma4ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = Gemma4ForConditionalGeneration.from_pretrained(
    "huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated", dtype="auto"
)
processor = AutoProcessor.from_pretrained(
    "huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated"
)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:20]")

def preprocess_function(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    return processor.apply_chat_template(
        messages, return_tensors="pt", padding=False, truncation=True,
        max_length=8192, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False,
    )

ds = ds.map(preprocess_function, batched=False, remove_columns=ds.column_names)

import torch
def data_collator(batch):
    assert len(batch) == 1
    return {
        key: (torch.tensor(value) if key != "pixel_values"
              else torch.tensor(value, dtype=torch.bfloat16).squeeze(0))
        for key, value in batch[0].items()
    }

oneshot(
    model=model, recipe=recipe, dataset=ds,
    max_seq_length=8192, num_calibration_samples=20,
    data_collator=data_collator,
)

model.save_pretrained("output-NVFP4", save_compressed=True)
processor.save_pretrained("output-NVFP4")

Environment

| Package | Version |

|---------|---------|

| torch | 2.11.0+cu130 |

| transformers | 5.5.4 |

| llmcompressor | 0.1.dev (main @ 3084520) |

| compressed-tensors | 0.15.1a20260414 |

| safetensors | 0.7.0 |

| CUDA | 13.0 |

Requirements

  • GPU: NVIDIA Blackwell (RTX 5090, RTX PRO 6000, B200, etc.) — NVFP4 requires SM 120
  • VRAM: ~16 GB minimum
  • Software: vLLM nightly (cu130 build), or any framework supporting compressed-tensors NVFP4

Notes

  • This is an abliterated (uncensored) model. The base model has had safety training removed. Use responsibly.
  • Vision tower is kept in BF16 — multimodal capabilities are preserved at full precision.
  • NVFP4 is a Blackwell-specific format. This checkpoint will not work on Ampere/Hopper GPUs.

Credits

Support the Base Model Author

If you find this model useful, please consider supporting huihui-ai — the creator of the abliterated base model: