中文Sign up

aday777/gemma-4-31B-it-abliterated-NVFP4

🤗 Hugging Face sourcetext-generationapache-2.031.3B params33 GBsafetensors✓ 2 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo aday777/gemma-4-31B-it-abliterated-NVFP4 ./model-folder
Needs a seeder →

Gemma-4-31B-it — Abliterated + NVFP4

An abliterated (refusal-direction removed) build of google/gemma-4-31B-it, quantized to plain NVFP4 (4-bit weights and activations) with llm-compressor / compressed-tensors, for native FP4 inference on NVIDIA Blackwell (sm_120) GPUs under vLLM.

⚠️ Safety notice. Abliteration removes the model's learned refusal behavior. This model will attempt to answer prompts the original -it model would decline. It is intended for research and use on your own hardware. You are responsible for how you use it; see Intended use & limitations below.

Why this build exists

At the time it was made, no public checkpoint combined all three of:

  1. abliterated / uncensored,
  2. plain NVFP4 (not NVFP4_AWQ, which stock vLLM rejects), and
  3. x86 Blackwell–runnable (not an ARM-only DGX Spark image).

Existing NVFP4 Gemma-4 builds were either the stock/censored model, the NVFP4_AWQ variant, or shipped only in ARM-only images. This repo fills that gap by self-quantizing a bf16 abliterate to plain NVFP4.

What was done (modifications from the base model)

This is a modified derivative of google/gemma-4-31B-it. Two changes:

  1. Abliteration — directional ablation / weight orthogonalization (Arditi et al., 2024). The refusal direction is estimated from mean last-token residual activations on matched harmful vs. harmless prompts, then orthogonalized out of every residual-writing weight (embed_tokens, per-layer attention o_proj and MLP down_proj). The ablation is baked into the weights — no inference-time hooks required.
  2. NVFP4 quantization — one-shot PTQ (E2M1 elements with FP8 block scales) on the text decoder's Linear layers only. The vision tower, multimodal projector, audio modules, token embeddings and the (tied) lm_head are left in higher precision. See recipe.yaml in this repo for the exact scheme and ignore list.

Hardware / software requirements

  • GPU: NVIDIA Blackwell with native FP4 tensor cores (sm_120, e.g. RTX PRO Blackwell). NVFP4 activation quantization needs hardware FP4 support.
  • Serving: a recent vLLM with compressed-tensors NVFP4 support.
  • ~20 GB on disk; weights fit comfortably on a single 24 GB+ card (leave KV-cache headroom), or use tensor parallelism.

Serving with vLLM (+ optional MTP speculative decoding)

Gemma 4 ships an official EAGLE/MTP-style draft, google/gemma-4-31B-it-assistant, which vLLM drives as a native multi-token speculator. Use "method": "mtp" (passing "draft_model" for a Gemma-4 assistant silently disables MTP):

vllm serve aday777/gemma-4-31B-it-abliterated-NVFP4 \
    --tensor-parallel-size 2 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.88 \
    --speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 3, "method": "mtp"}' \
    --port 8003

Drop the --speculative-config line to serve without speculative decoding.

Intended use & limitations

  • Intended use: local research, red-teaming, evaluation, and applications where you supply your own guardrails.
  • No safety filtering: refusal behavior has been removed; this model can produce harmful, offensive, or otherwise objectionable content. It is not suitable for unsupervised or public-facing deployment without your own safety layer.
  • Quantization: 4-bit weights and activations trade some quality for speed and memory; expect small accuracy differences from the bf16 model.
  • All original capability limitations of gemma-4-31B-it still apply.

License & attribution

Derived from google/gemma-4-31B-it by Google DeepMind, licensed under the Apache License 2.0 (see Gemma 4 license). This derivative is distributed under the same license; a copy of the Apache 2.0 License is included as LICENSE. Per the license, note that these files have been modified from the original (abliterated and NVFP4-quantized as described above). Please also review Google's Gemma prohibited-use policy.

Reproduction

The model was produced with directional-ablation + llm-compressor NVFP4 PTQ. The exact quantization recipe is in recipe.yaml. Abliteration calibration used matched harmful/harmless instruction sets and selected the refusal direction by the layer that most reduced refusals on a validation split.

Support

If this model is useful to you, Bitcoin donations are welcome:

bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p