RedHatAI/gemma-4-31B-it-speculator.dflash2

Model Overview

• Model Architecture: Gemma4ForConditionalGeneration
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Speculative Decoding Algorithm: DFlash2
  • Draft Model: 5-layer qwen3 backbone (DFlash2DraftModel)
• Release Date: 2026-10-14
• Version: 1.0
• Model Developers: RedHatAI

A DFlash2 speculative-decoding draft model for google/gemma-4-31B-it, trained with Speculators.

Model Specifications

Training Details

Trained with the Speculators library (ref 442ce1bf4e2f70dd0b45f36e63bd5849fbb35912, vLLM base image v0.29.0) on regenerated google/gemma-4-31B-it responses over mlabonne/open-perfectblend prompts.

Full cluster training recipe (YAML)

metadata:
  name: gemma4-31b-dflash2-bs8

images:
  builds:
    shared:
      vllm_base: docker.io/vllm/vllm-openai:v0.29.0
      speculators_ref: "442ce1bf4e2f70dd0b45f36e63bd5849fbb35912"

extractor:
  replicas: 1
  tensor_parallel_size: 2
  data_parallel_size: 2
  trust_remote_code: true
  max_model_len: 32000
  resources:
    gpu_count: 4
    node_selector:
      nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
  vllm_args: ["--gpu-memory-utilization", "0.92"]

data_preparation:
  defaults:
    sources:
      - hf:nm-research/gemma-4-31b-it-responses-open-perfectblend

trainers:
  defaults:
    nodes: 1
    resources:
      gpu_count: 4
      node_selector:
        nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
    config:
      speculator_type: dflash2
      verifier:
        verifier_name_or_path: google/gemma-4-31B-it
        trust_remote_code: true
      draft:
        draft_vocab_size: null
        num_layers: 5
        target_layer_ids: [1, 17, 29, 47, 59]
        sliding_window: 2048
      data:
        hidden_states_backend: mooncake
        total_seq_len: 8192
        num_workers: 2
        prefetch_factor: 1
        max_anchors: 1024
      generation:
        on_missing: generate
        on_generate: delete
        request_timeout: 180
        max_retries: 20
        max_consecutive_generation_failures: 20
      loss:
        loss_fn: '{"ce":0.1,"tv":0.9}'
      optimizer:
        optimizer: muon
        lr: 6e-4
        weight_decay: 0.01
        muon_lr: 0.001
      scheduler:
        scheduler_type: cosine
      trainer:
        epochs: 2
        checkpoint_freq: 0.1
        log_freq: 100
        fsdp_shard: true
        gradient_checkpointing: false
      dflash:
        block_size: 8

Deployment

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 4 \
  --speculative-config '{
    "model": "RedHatAI/gemma-4-31B-it-speculator.dflash2",
    "num_speculative_tokens": 7,
    "draft_tensor_parallel_size": 4,
    "method": "dflash"
  }'

Use the verifier's chat template via the /chat/completions endpoint.

Evaluation

Evaluated with Speculators 0.8.0 on vLLM 0.29.0 against the RedHatAI/speculator_benchmarks dataset (evaluate.py ... throughput, 200 requests, max concurrency 16, temperature 0, enable_thinking: true).

Per-position token acceptance rates across datasets:

Performance benchmarking

Throughput vs. interactivity, comparing the google/gemma-4-31B-it verifier with no speculator against the verifier served with this DFlash2 draft model (7 speculative tokens), at matched request concurrency (N=1,2,4,8,16,32,64,128):

At the highest-throughput point measured (N=128 concurrent requests):

Configuration and command

Configuration

• temperature: 0
• max output tokens: 1024
• concurrency (N): 1, 2, 4, 8, 16, 32, 64, 128
• repetitions: 1 run per point
• vLLM version: 0.29.0
• GuideLLM version: 0.7.4
• Verifier served with: --tensor-parallel-size 4 --gpu-memory-utilization 0.90 --max-model-len 8192 --max-num-seqs 256 --max-num-batched-tokens 16384 --no-enable-prefix-caching

Command

throughput_interactivity.py collect \
  --target http://localhost:8120 \
  --model google/gemma-4-31B-it \
  --tokenizer google/gemma-4-31B-it \
  --dataset RedHatAI/speculator_benchmarks \
  --subset HumanEval \
  --max-tokens 1024 \
  --repeats 1 \
  --extra-body temperature=0 \
  --streams 1,2,4,8,16,32,64,128

Optimization

This model implements DFlash2 speculative decoding: a 5-layer draft model proposes up to 7 tokens per step, which the google/gemma-4-31B-it verifier accepts or rejects in parallel. The average accepted length in the table above (4.0–6.9 tokens, depending on subset) quantifies the expected decode-throughput speedup — confirmed directly by the measured 1.5x+ throughput gains above.

License

This model is distributed under the Apache-2.0 license. The Gemma-4 verifier is subject to its own license and terms.