Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4

A 6.2 GB multimodal (vision + text + audio) checkpoint that fits on a single RTX 5090 and decodes at ~516 tok/s (long-form decode, MTP K=5, same harness as the W4A4 revision) with native Blackwell FP4 (W4A4) + FP8 e4m3 embeddings + MTP speculative decoding.

Name breakdown: Gemma-4-E4B (base) → Uncensored-HauhauCS-Aggressive (base fine-tune) → MTP (native multi-token-prediction drafter support, --spec-tokens 5 recommended) → NVFP4 (W4A4 ModelOpt quantization of the text transformer: 4-bit weights and 4-bit activations — the format that runs on Blackwell's native FP4 tensor cores).

This is a community-built, hand-assembled checkpoint: the HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive fine-tune (GGUF) was converted to HF format, quantized to NVFP4 W4A4 with NVIDIA ModelOpt, and re-assembled with the base model's multimodal encoders (vision tower + audio encoder kept in BF16).

It is not an official Google or HauhauCS release. It exists because nobody had published a native multimodal NVFP4 build of this model that serves with stock vLLM on consumer Blackwell.

What's in this revision (vs the W4A4 revision)

This revision replaces the text-transformer embeddings (embed_tokens, the tied lm_head, and the per-layer embedding table embed_tokens_per_layer) with FP8 e4m3 + per-row F32 scales. Everything else — the W4A4-quantized 42 text layers, the BF16 vision tower and audio encoder, the tokenizer, the configs — is byte-identical to the W4A4 revision of this repo.

Measured on the same hardware, same time window, same harness (RTX 5090 shared GPU, vLLM 0.28, MTP K=5, greedy, 2 runs × 6 fixed prompts, streaming with include_usage):

Decode speed is flat (within run-to-run noise) — the FP8 conversion buys memory, not speed: a smaller resident PLE table leaves ~2.6 GiB for KV cache, which at gpu-memory-utilization 0.40 (the shared-GPU setting) is +134 801 tokens of KV cache (260 888 vs 126 087, a 2.07× increase). On a dedicated GPU you can raise --max-model-len to the full 131 072 window and still keep the KV cache the W4A4 revision had at 42 000.

Behavior on the 8-prompt greedy battery (fact traps + controlled questions): 1/8 byte-identical to the W4A4 revision, 1/8 byte-identical to the W4A16 revision, while the W4A4 vs W4A16 reference pair is 0/8 — i.e. FP8-embeds sits between the two revisions on the same confidence axis. Fact traps are answered the same; post-cutoff traps hedge slightly more than W4A4, less than W4A16. No new error categories appeared.

Why the checkpoint needs a patch to serve

vLLM 0.28's VocabParallelEmbedding has no native path for quantized embeddings — it assumes the loaded weight is the dtype the forward pass computes in. This checkpoint stores the two embedding tables as FP8 e4m3 with per-row F32 scales (2 bytes per weight → 1 byte per weight, plus a 4-byte scale per row). Serving the checkpoint unpatched loads the FP8 tables and casts them to BF16 at load time with no rescale, so every embedding is ~448× too small and the model degenerates (repeated <unused53> + EOS).

sitecustomize.py in this repo fixes that on PYTHONPATH:

• embed_tokens_per_layer (262 144 × 10 752, the PLE table) stays resident in VRAM as FP8 e4m3. The patched VocabParallelEmbedding.forward gathers the selected rows as FP8 (110 KB/token), casts to BF16, multiplies by the per-row scale, and views as (n, dim). Bit-identical to loading the BF16 table.
• embed_tokens (262 144 × 2 560, tied to lm_head) is dequantized to BF16 once, in-place, at process_weights_after_loading. Doing the per-row dequant on the forward path for the tied lm_head would add a per-token F8→BF16 gather to the logits path for no gain — this table is only ever read once per token as a scatter-lookup, so a one-shot BF16 copy is the right call.
• The MTP drafter (google/gemma-4-e4b-it-assistant) carries its own BF16 embed_tokens (262 144 × 256). The patch marks parameters by data_ptr at weight_loader time and only processes parameters that received FP8 data, so the drafter's BF16 table is skipped — leaving it untouched is what keeps MTP acceptance at the W4A4-revision level instead of collapsing to 0/15.

Verified on vLLM 0.28, Python 3.12, CUDA 13, RTX 5090 (32 GB, shared): patch logs F8 load into param (262144, 10752) + F8 load into param (262144, 2560), then PLE F8-resident (262144,10752) scale_max=0.002563 and small table dequanted (262144,2560) scale_max=0.001238. KV cache goes from 45 573 tokens (unpatched, 1.09x concurrency at 42 000) to 260 888 tokens (6.21x) at the same gpu-memory-utilization 0.40 — the 2.62 GiB of freed PLE VRAM plus the FP8-resident tables translates directly into a 2.07× larger KV cache.

What's inside

Files: single model.safetensors (6.2 GB, 3,375 tensors — two of them FP8 e4m3 with accompanying weight_scale tensors) + tokenizer/processor/config + sitecustomize.py (required for serving).

Serve with vLLM (W4A4 + B12X kernel — the fast path)

Verified on vLLM 0.28, Python 3.12, CUDA 13, RTX 5090 (32 GB, shared). W4A4 activates the native FP4 tensor cores via the B12X GEMM kernel, which requires the b12x extra:

# One-time venv setup
uv pip install 'vllm==0.28.0' b12x==1.2.4
uv pip install apache-tvm-ffi==0.1.11   # THE pin — see gotcha below

# Fetch the serving patch from the repo (this revision's version merges the
# B12X pre-import fix AND the FP8-embedding load/forward fix in one file)
mkdir -p ~/.vllm-patches && cd ~/.vllm-patches
wget https://huggingface.co/Adolphsson/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4/resolve/main/sitecustomize.py

# Serve
export CUDA_HOME=$VENV/lib/python3.12/site-packages/nvidia/cu13
export PATH=$VENV/bin:$VENV/lib/python3.12/site-packages/nvidia/cu13/bin:$PATH
export LD_LIBRARY_PATH=$VENV/lib/python3.12/site-packages/nvidia/cu13/lib:$LD_LIBRARY_PATH
export FLASHINFER_CUDA_ARCH_LIST=12.0
export CCCL_DISABLE_CTK_COMPATIBILITY_CHECK=1
export PYTHONPATH=$HOME/.vllm-patches    # the sitecustomize.py from the repo

vllm serve Adolphsson/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive-MTP-NVFP4 \
  --linear-backend b12x \
  --port 8000 --host 0.0.0.0 \
  --served-model-name gemma4-e4b-nvfp4 \
  --max-model-len 42000 \
  --gpu-memory-utilization 0.40 \
  --max-num-seqs 2 \
  --limit-mm-per-prompt '{"image":4,"video":2,"audio":4}' \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --enable-prefix-caching \
  --mm-processor-cache-gb 2 \
  --spec-model google/gemma-4-e4b-it-assistant \
  --spec-tokens 5

VRAM footprint (measured, RTX 5090 / 32 GB)

--gpu-memory-utilization is a fraction of the card's 32 GB. Of that budget, ~7.5 GB is fixed cost (6.2 GB weights + runtime + CUDA graphs) regardless of setting — everything above that becomes KV cache. That's why the KV-cache column jumps faster than the utilization figure: it scales with the absolute budget, not the percentage.

Measured by booting the server at each setting and reading the vLLM process's own VRAM via nvidia-smi --query-compute-apps (excluding other tenants on the card).

• 0.28 is the minimum that boots a 42K-context server — 0.26 OOMs ("0.82 GiB needed, 0.38 GiB available"). It is a knife-edge: the model + a 42K ceiling just fits. For anything more than short assistant turns, 0.40 is worth the ~4 GB: 4× the KV cache.
• For the full 131K native window, use 0.72 → ~22.5 GB, 839K-token KV cache.

Fitting a full voice pipeline on one 5090. If the model's built-in audio isn't enough and you want to run external STT + this LLM + a high-quality low-latency TTS (e.g. OmniVoice) on the same card: the 0.28 setting holds the model to ~9.6 GB, leaving ~22 GB free of the 32 GB — comfortable headroom for an fp16 Whisper-class STT and an fp16 OmniVoice-class TTS alongside it, all on one card, minimal latency.

With the av (PyAV) + soundfile packages installed in the venv, all three modalities are accepted at the API level (pip install av soundfile).

B12X gotchas (three of them, all hit, all documented)

1. apache-tvm-ffi must be exactly 0.1.11. The b12x package transitively drags in apache-tvm-ffi 0.1.13.post3, which double-registers the __ffi_repr__ type attribute and crashes the engine after CUDA-graph capture with a C++ abort (tvm::ffi::Error: TypeAttr __ffi_repr__ is already registered for type index 132). vLLM 0.28 pins 0.1.11 — respect the pin.
2. CCCL toolkit compatibility check. With the nvidia/cu13 venv layout, FlashInfer's JIT compiles against nvcc 13.3 while the bundled headers are 13.0; CCCL's cuda_toolkit.h aborts on the minor mismatch. The CCCL_DISABLE_CTK_COMPATIBILITY_CHECK=1 env var is not read by the C preprocessor, so patch the header in the venv: in .../flashinfer/data/cccl/libcudacxx/include/cuda/std/__cccl/cuda_toolkit.h, change the guard #ifndef CCCL_DISABLE_CTK_COMPATIBILITY_CHECK to #if 0.
3. torch.compile fullgraph vs lazy b12x imports. vLLM 0.28's B12xNvFp4LinearKernel.apply_weights() calls importlib.import_module inside the traced path, which fullgraph forbids (Unsupported: function marked as skipped). Fix: a sitecustomize.py on PYTHONPATH that pre-imports the b12x submodules and rebinds the kernel module's accessor functions to direct module references (a MetaPathFinder intercepting vllm.utils.b12x post-execution).

If you would rather skip all three, the fallback is the W4A16 variant (previous revision of this repo) served with --quantization modelopt_fp4 and no --linear-backend flag — same model, ~27% slower decode (see Benchmarks).

One patch required for audio input (vLLM 0.28 + transformers 5.6)

The quantized audio encoder hits a non-contiguous tensor in Gemma4ClippedLinear. In transformers/models/gemma4/modeling_gemma4.py, change the Gemma4ClippedLinear.forward line:

hidden_states = self.linear(hidden_states.contiguous())   # was: self.linear(hidden_states)

This is the only quantized multimodal tower (vision/embeddings are excluded from NVFP4), which is why image input works unpatched but audio input crashes without it.

MTP speculative decoding (~2x decode speed)

This model ships with native MTP support in vLLM 0.28 (Gemma4MTP):

  --spec-model google/gemma-4-e4b-it-assistant \
  --spec-tokens 5

Measured on the RTX 5090, vLLM 0.28 (fresh server per K, 300-token cap, greedy, streaming with include_usage, 2 runs × 6 prompts). Both revisions measured in the same session, same GPU, same harness — directly comparable:

All figures in tok/s. "long-form" = token-weighted average of runs ≥300 tokens (story/long/math prompts); "mixed" = all runs including short-answer prompts. Acceptance = accepted_tokens / draft_tokens from vllm:spec_decode_* Prometheus counters.

W4A4's native FP4 GEMM is the single largest decode-speed win available on this hardware — the activation quantization removes the bf16 dequant/multiply penalty that weight-only NVFP4 carries. Note the win is MTP-dependent: without the drafter, both revisions decode at ~242 tok/s (flat). The FP4 activation gain pays off in the compute-bound draft-verify path, where the speedup is 2.1× (K=5 vs K=0).

FP8-embeds matches or slightly exceeds W4A4 across all K — the FP8 embedding tables are the same size per token as BF16 (after per-row dequant in forward), so decode speed is identical. The KV-cache gain (2.07×) is the practical benefit on shared GPUs.

K=5 is the recommendation for shared GPUs. The acceptance curve is monotonically decreasing in K (0.67 at K=1 → 0.30 at K=5) but the net speed keeps rising because each iteration proposes more tokens than it wastes — diminishing returns above K=5 (the old revision's K=6 data showed +2% over K=5, not worth the extra drafter re-execution on shared hardware).

Previous revision (W4A4 only, old wall-clock harness, 600-token cap)

For reference, the W4A4 revision (this repo, earlier commit) was benchmarked with a different harness (non-streaming wall-clock, 600-token cap, 4 prompts):

These are the figures cited in the W4A4 revision's README. The 600-token wall-clock harness reports slightly higher numbers than the 300-token streaming harness (longer runs amortize the prefill better; the wall-clock metric includes prefix-cache hits). Both harnesses agree on the relative K=0 vs K=5 speedup (~2.1–2.6×).

Quick test

curl http://localhost:8000/v1/chat/completions -d '{
  "model": "gemma4-e4b-nvfp4",
  "messages": [{"role":"user","content":"Write a short poem about cats"}],
  "max_tokens": 300, "temperature": 0.7
}'

Multimodal (vision) via the OpenAI-compatible input_image content part; audio via input_audio (see known issue below).

Known issues (be honest section)

1. sitecustomize.py is required on PYTHONPATH (this revision only). The FP8 embedding tables are loaded through a patched VocabParallelEmbedding (see "Why the checkpoint needs a patch to serve" above). Without it the engine comes up, health returns 200, but the model degenerates: repeated <unused53> + immediate EOS, and the KV cache is ~4× smaller because the FP8 tables get cast to BF16 at load with no rescale. Two failure modes I hit myself, both silent: (a) PYTHONPATH pointing at a directory whose sitecustomize.py is a different file — Python imports only the first sitecustomize.py on the path, so PYTHONPATH=/root/b12x:/root/fp8embed ran the old B12X-only patch and the FP8 fix never loaded; (b) the drafter's own BF16 embed_tokens matching the target's shape — the patch resolves this by marking parameters by data_ptr at weight_loader time and only processing parameters that received FP8 data, so the drafter is skipped and MTP acceptance stays at the W4A4-revision level instead of collapsing to 0/15.
2. Audio input requires the 1-line patch (fixed, documented above): Gemma4ClippedLinear.forward in transformers' gemma4 module — add .contiguous() before the quantized linear. Without it the engine dies on the first input_audio request. Verified on the W4A16 revision (transcription, name extraction, word counting all work); the W4A4 text path is kernel-invariant for the audio tower (still BF16), but re-verify on your setup.
3. FP8 embeddings hedge slightly more than the W4A4 revision did. Same measurement as the previous revision's confidence note: the 8-prompt greedy battery is 1/8 byte-identical between the W4A4 and FP8-embeds revisions (the W4A4 vs W4A16 reference pair is 0/8, so 1/8 is "much closer to W4A4 than to anything else" rather than "identical"). Where the W4A4 revision stated an uncertain post-cutoff fact, the FP8-embeds revision tends toward the hedge. Controlled fact questions agree exactly across all three revisions. The per-row dequant is bit-identical to a BF16 load, so the difference is the FP8 mantissa (3-bit) on the embedding rows, not the patch. If you want the previous revision's exact character, check out its git revision on this repo — the weights are one file apart.
4. Uncensored fine-tune behavior: inherits the HauhauCS-Aggressive training profile (direct, less cautious phrasing). Same model, quantized — expect the same personality as the GGUF, plus the confidence shifts noted in items 3 and the previous revision's item 2.
5. NVFP4 calibration: quantized with ModelOpt NVFP4_DEFAULT_CFG (max algorithm), calibration on 24 self-generated sequences covering the model's own Swedish/English/code/math output distribution; text quality is excellent in practice, but per-linear weight_scale_2 divergence across fused QKV is present (vLLM logs a warning) — typical for NVFP4 Gemma-class models. The FP8 embedding conversion used the deterministic per-row rule scale = row_max / 448 (448 = F8 e4m3 max), no calibration data.
6. Reasoning channel is raw-text, not structured: with chat_template_kwargs: {"enable_thinking": true} the model produces a full reasoning channel, measured at ~470 tok/s on the W4A4 revision. The W4A16 revision's 624–700 tok/s figure did not reproduce — but an A/B test (same prompts, same greedy settings, acceptance read from vllm:spec_decode_* counters) showed drafter acceptance is identical between the revisions (0.520 on W4A4 vs 0.514 on W4A16) and thinking repetition is identical (distinct-5 n-gram diversity 0.973 on both, i.e. the thought stream is highly varied, not looping, in either format). The FP4 activation quantization therefore does not measurably change the drafter's ability to predict thought tokens. The lower wall-clock figure is most likely a measurement artifact of the older harness (per-delta counting on a stream that chunks differently) plus prompt-length differences, not a drafter regression. Streaming caveat: vLLM 0.28's gemma4 thinking path delivers the entire completion as a single SSE content chunk (one delta, then finish) instead of per-token deltas. Wall-clock throughput is unaffected, but clients that count or animate per delta will see one giant chunk — render on a timer, not on delta count. How it is consumable per interface:
   • Chat API + streaming: the thought text arrives as plain content (one big delta, with a literal "thought\n" prefix to strip). Works in real time — the recommended path for voice/latency pipelines, but see the streaming caveat above.
   • Chat API, non-streaming: both reasoning and content come back empty — the parser discards the unclosed thought channel.
   • /v1/completions: full raw text in text, fully consumable.
   • The reasoning field is never populated (the uncensored fine-tune emits thought-channel markers the vLLM 0.28 gemma4 parser doesn't map to structured fields). Clients that render thinking in a separate UI panel get an empty panel — consume content and strip the thought prefix instead.

Benchmarks & provenance

Raw JSON results live in the author's notes linked below. Methodology: same GPU, same time window, 2 runs per configuration, greedy, 4 fixed prompts, 600-token cap.

• Built: August 2026, on an RTX 5090 LXC container
• W4A4 re-quantization: August 2026, ModelOpt 0.46.0 NVFP4_DEFAULT_CFG, calibration on 24 self-generated sequences (Swedish chat, English technical, code, math)
• FP8 embedding conversion (this revision): September 2026, deterministic per-row scale = row_max / 448 (F8 e4m3 max), applied to embed_tokens and embed_tokens_per_layer only — all other tensors byte-identical to the W4A4 revision
• Benchmark harness (this revision): 2 runs × 6 fixed prompts (3 short, 3 long-form, 300-token cap), greedy, fresh server per K, streaming with include_usage (token count from the final usage chunk — vLLM chunks multiple tokens per SSE event under MTP, so per-delta counting undercounts by ~2.9× at K=5; fixed in this revision's harness), acceptance from vllm:spec_decode_* Prometheus counters — same harness run against both the W4A4 and FP8-embeds revisions on the same GPU in the same session
• Base fine-tune: HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive (GGUF Q8_K_P)
• Encoders/config: google/gemma-4-e4b
• Serving: vLLM 0.28, --linear-backend b12x (W4A4 native FP4) + sitecustomize.py from this repo on PYTHONPATH
• MTP drafter: google/gemma-4-e4b-it-assistant (separate, 159 MB)

License

Lineage

• Original base model: google/gemma-4-e4b-it
• Source model: HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive
• Direct contribution: NVFP4 W4A4 quantization, verified B12X native-FP4 serving, verified MTP speculative decoding compatibility, and Blackwell-oriented optimization (previous revision); FP8 e4m3 embedding conversion with per-row scales plus the sitecustomize.py serving patch (this revision)

This model is a technical optimization/conversion of the upstream HauhauCS model. The uncensored behavior was created by HauhauCS, not by this release.

Modifications

• NVFP4 W4A4 quantization (4-bit weights + activations) using NVIDIA ModelOpt
• Verified B12X native-FP4 GEMM serving on SM120 (RTX 5090)
• Verified compatibility with Gemma 4 E4B MTP speculative decoding
• Blackwell-oriented optimization
• FP8 e4m3 conversion of the two text-embedding tables with deterministic per-row F32 scales, plus the sitecustomize.py serving patch that loads them (this revision)

Gemma 4 is Apache-2.0 licensed. This release is based on the Gemma 4 E4B model and the HauhauCS derivative described above. Please refer to the upstream model for its original licensing and attribution information. Full license text: LICENSE · Upstream attribution: NOTICE