Sign up

Reza2kn/gemma-4-26B-A4B-it-assistant-NVFP4

🤗 Hugging Face sourcetext-generationapache-2.0268 MBsafetensors✓ HF checksums availableupdated today
No torrent yet →

Gemma 4 26B A4B IT Assistant MTP Draft - NVFP4

NVFP4 quantization of google/gemma-4-26B-A4B-it-assistant, the MTP/draft assistant model.

This is a quantized derivative, not a fine-tune. The repo metadata sets base_model_relation: quantized and tags the base model accordingly.

Format

  • 2D BF16 weight tensors are stored as packed NVFP4 E2M1 codes.
  • Per-block scales use FP8 E4M3, one scale per 16 values.
  • 1D norm/scalar tensors remain BF16.
  • The tied embedding/head tensor is quantized; there is no separate lm_head.weight in the source checkpoint.

Because this Gemma 4 assistant architecture currently requires source/newer Transformers support, the repo includes load_gemma4_nvfp4.py, which dequants the packed NVFP4 tensors into the upstream Gemma4AssistantForCausalLM module.

from load_gemma4_nvfp4 import load_model

model, tok = load_model("Reza2kn/gemma-4-26B-A4B-it-assistant-NVFP4", device="cuda")
prompt = "Explain in one sentence what a draft model does."
inputs = tok(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Files

  • nvfp4_model.safetensors - packed NVFP4 weights plus BF16 residual tensors.
  • quantization_config.json - tensor map, block size, parameter counts, and format metadata.
  • quant_error_report.json - per-tensor relative L2 quantization error.
  • load_gemma4_nvfp4.py - loader/smoke-test helper.

Notes

This is a storage-format quantization for the new Gemma 4 assistant draft architecture. Native NVFP4 kernel acceleration depends on runtime support catching up to this architecture; the included loader provides a correctness-first dequant path.

Licensing and attribution

This model distribution is licensed under the Apache License, Version 2.0. See LICENSE. Existing third-party copyright, license, and attribution notices remain applicable.