Gemma 4 26B-A4B — MXFP8 + FP8 KV + Static FP8 Attention Metadata
This checkpoint is published as intel-ai/gemma-4-26B-A4B-MXFP8-FP8KV-FP8Attn-CT-AutoRound and is derived from
google/gemma-4-26B-A4B.
It retains the source model's text and vision components and is serialized in
compressed-tensors format. Routed-expert weights and source-present text
self-attention projections use MXFP8; the router, shared MLP, vision tower,
embeddings, and other retained weights remain BF16. The checkpoint also
contains static FP8 KV and attention metadata.
Scope and runtime caveats: The evaluation configured
max_model_len=131072, but did not use 128K-token prompts. It is not a long-context stress test. vLLM logs confirm the FP8 KV-cache dtype andTRITON_ATTNbackend; they do not establish that the checkpoint's static KV scales or static FP8 attention metadata were consumed. The results below cover text tasks only and are not a general quality guarantee.
1. Model summary
| Property | Value |
|---|---|
| Architecture | Gemma4ForConditionalGeneration |
| Text transformer structure | 30 layers; 128 routed experts per layer; top-8 routing |
| Attention layout | 25 sliding-attention layers and 5 full-attention layers |
| Weight formats | MXFP8 (F8_E4M3) with BF16-retained modules |
| KV cache | FP8 scheme and scale metadata included in the checkpoint |
| Attention metadata | Static FP8 attention metadata included; runtime use not established |
| Serialization | compressed-tensors |
| Evaluation context limit | 131,072 tokens (max_model_len) |
| Checkpoint size | 28,415,419,904 bytes across six safetensors shards |
Weight and scale inventory
| Tensor group | Stored dtype | Tensor count | Weight elements |
|---|---|---|---|
| Routed-expert weights | F8_E4M3 |
11,520 | 22,837,985,280 |
| Source-present text self-attention projections | F8_E4M3 |
115 | 1,110,179,840 |
| Retained weights, including router, shared MLP, vision, and embeddings | BF16 |
838 | 1,857,771,086 |
| Total model-weight tensors / elements | Mixed | 12,473 | 25,805,936,206 |
| MXFP8 block-scale tensors (metadata) | U8 |
11,635 | — |
| Scale and metadata tensors | F32 |
171 | — |
Scale tensors are listed separately and are not included in the model-weight element total. The vision component is retained, but vision quality was not measured in the evaluation below.
Exported quantization settings
| Component | Recorded setting |
|---|---|
| Weights | 8-bit floating point (F8_E4M3), symmetric, static, group-wise with group size 32 |
| Weight scales | U8 block/group scale tensors |
| Input-activation metadata | 8-bit float, symmetric, dynamic and group-wise with group size 32; exported metadata only |
| KV-cache scheme | 8-bit float, symmetric, static, tensor-wise |
| Attention input metadata | 8-bit float, symmetric, static, tensor-wise; targets include Gemma4TextAttention and Gemma4VisionAttention |
The configuration records activation and attention metadata; its presence in the checkpoint does not imply that a given inference runtime applies it. See the runtime caveats below.
Command-line arguments used for this checkpoint
The following are the quantization arguments used to create this artifact. Source and output paths are placeholders rather than machine-specific paths.
--model_name <source-model-directory>
--scheme MXFP8
--algorithm rtn
--iters 0
--static_kv_dtype fp8
--static_attention_dtype fp8
--ignore_layers router,vision_tower,embed_vision,embed_tokens,mlp
--device_map 0,1,2
--format llm_compressor
--output_dir <output-directory>
--device_map refers to process-visible GPU indices; expose three suitable
idle devices before using the listed mapping.
2. Evaluation results
Measured on 2026-09-30 with lm-evaluation-harness 0.4.13 and vLLM 0.29.0.
Scores use the primary metric for each task; GSM8K uses strict-match.
| Benchmark | Metric | Score | Samples |
|---|---|---|---|
| PIQA | accuracy | 82.75% | 1,838 |
| MMLU | accuracy | 74.41% | 14,042 |
| HellaSwag | accuracy | 63.40% | 10,042 |
| GSM8K | strict exact match | 73.69% | 1,319 |
| Unweighted arithmetic mean | — | 73.57% | — |
Tested runtime configuration
| Setting | Value |
|---|---|
| Hardware | 2 × NVIDIA GeForce RTX 5090 |
| Attention backend | TRITON_ATTN |
| Parallelism | tensor parallel 2, pipeline parallel 1 |
| lm-eval batch size | 64 |
| vLLM scheduling limit | max_num_seqs=64 |
| Compute dtype / KV cache | BF16 / FP8 |
| Context limit | max_model_len=131072 |
| Other evaluation settings | text-only; thinking disabled; seed 42; 2,048-token generation limit |
| End-to-end evaluation wall time | 1,393.32 seconds (23:13.32); not a latency or throughput measurement |
BF16 + BF16 KV baseline comparison
The BF16 + BF16 KV baseline values below were supplied for this comparison and
were not rerun for this README update. Candidate values are from the MXFP8 +
FP8 KV evaluation above. Scores and ratios are shown to two decimal places;
MXFP8 / BF16 is the candidate score divided by the BF16 baseline score,
expressed as a percentage. The score difference is in percentage points.
| Benchmark | Metric | Samples | BF16 + BF16 KV (baseline) | MXFP8 + FP8 KV | Δ (pp) | MXFP8 / BF16 |
|---|---|---|---|---|---|---|
| PIQA | accuracy | 1,838 | 82.32% | 82.75% | +0.44 | 100.53% |
| MMLU | accuracy | 14,042 | 74.40% | 74.41% | +0.01 | 100.02% |
| HellaSwag | accuracy | 10,042 | 63.32% | 63.40% | +0.08 | 100.13% |
| GSM8K | strict exact match | 1,319 | 74.37% | 73.69% | -0.68 | 99.08% |
| Unweighted arithmetic mean | — | — | 73.60% | 73.57% | -0.04 | 99.95% |
Displayed scores, deltas, and ratios are independently rounded to two decimal places; deltas and ratios are calculated from the unrounded scores.
This is a descriptive comparison, not a controlled weight-only quantization
test: the baseline uses BF16 KV cache while the candidate uses FP8 KV cache,
and the candidate was measured with TRITON_ATTN, TP=2/PP=1, and batch size
64. The ratios and deltas should not be interpreted as isolating the effect
of MXFP8 weights. The means are unweighted averages of the four task scores;
the mean ratio is the ratio of those two means.
3. Reproducing the text evaluation
The following commands mirror the tested lm-eval settings. Select two available GPUs with sufficient memory before running; other hardware and parallel layouts have not been validated.
MODEL_ID="/path/to/downloaded/checkpoint"
MODEL_ARGS="pretrained=${MODEL_ID},tensor_parallel_size=2,pipeline_parallel_size=1,max_model_len=131072,gpu_memory_utilization=0.85,dtype=bfloat16,trust_remote_code=True,add_bos_token=True,enable_prefix_caching=False,max_gen_toks=2048,attention_backend=TRITON_ATTN,language_model_only=True,kv_cache_dtype=fp8,max_num_seqs=64,enable_thinking=False"
lm_eval --model vllm --model_args "$MODEL_ARGS" \
--tasks piqa,mmlu,hellaswag --batch_size 64 --log_samples --seed 42
lm_eval --model vllm --model_args "$MODEL_ARGS" \
--tasks gsm8k --batch_size 64 --log_samples --seed 42 --num_fewshot 5
4. Runtime notes and limitations
- The tested runtime selected FP8 KV-cache storage and
TRITON_ATTN. The logs did not confirm that the checkpoint's static KV scale tensors were consumed, so the configured cache dtype should not be taken as proof that those scales were applied. - Static FP8 attention metadata is present in the checkpoint, but its use by vLLM 0.29.0 was not verified. These results are not evidence that FP8 attention executed.
- The benchmarks are text-only. Vision or other multimodal prompts, thinking-on behavior, 128K prompt quality, long-context retrieval, and serving latency/throughput were not measured.
- Performance and quality on other GPUs, vLLM versions, backends, and parallel layouts are unverified.
5. License and attribution
This checkpoint is derived from Google's Gemma 4 26B-A4B and is not an official Google release. The upstream model metadata lists Apache 2.0. Review the Gemma 4 license and comply with its terms when using or redistributing this checkpoint.