Gemma-4 26B-A4B DSpark — Speculative Decoding Draft for vLLM

DSpark draft model for google/gemma-4-26B-A4B-it. Speeds up Gemma 4 26B-A4B inference with DSpark speculative decoding in vLLM — up to ~2× output throughput vs the same target without a draft (math, concurrency 1).

This checkpoint is a draft head only (not a standalone LLM). Pair it with the Gemma-4 26B-A4B-it target.

[!IMPORTANT] This model cannot generate text on its own. It needs intermediate hidden states from the 26B target at every forward pass. AutoModel.from_pretrained loads the weights, but this repo has no standalone generate() path.

Quick start (vLLM)

vllm serve google/gemma-4-26B-A4B-it \
  --speculative-config '{"method": "dspark", "model": "systalyze/gemma4-26b-a4b-it-dspark", "num_speculative_tokens": 7}'

Benchmarks vs base (no draft)

Same GPUs, same prompts: base = target alone; +DSpark = this draft + target.
AL = (accepted + num_drafts) / num_drafts. aiperf 0.12, 512 requests, temperature 1.0, ignore_eos=true.

Coding throughput

ISL ~U[3500,10000], OSL ~U[200,400].

Speedup by domain

ISL/OSL held fixed; only prompt content changes.

Acceptance length by domain

Context length (coding, OSL fixed)

AL stays flat as context grows; speedup falls mainly from target verify cost.

Speedup vs base

Acceptance length

Architecture

The tokenizer is copied from google/gemma-4-26B-A4B-it; the draft shares the target vocabulary.

Intended use

Draft component for speculative decoding of google/gemma-4-26B-A4B-it. With an exact verifier, accepted tokens match the target distribution — the draft affects latency/throughput, not response content.

Out of scope: standalone generation, classification, embeddings, or pairing with any other target.

Loading with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "systalyze/gemma4-26b-a4b-it-dspark",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="flex_attention",
)
tokenizer = AutoTokenizer.from_pretrained(
    "systalyze/gemma4-26b-a4b-it-dspark",
    trust_remote_code=True,
)

trust_remote_code=True and attn_implementation="flex_attention" are required (flex BlockMask attention path). Hub task is text-generation (not feature-extraction) — this is still a draft head for speculative decoding, not a standalone generator.

Training

Distilled against google/gemma-4-26B-A4B-it (thinking disabled) on prompts from Open PerfectBlend with responses regenerated by the target. Training used cached target hidden states.

Files

modeling_gemma4_dspark.py, configuration_gemma4_dspark.py, common.py, markov_head.py, sampling.py, and metrics.py are vendored from DeepSpec (relative imports + Gemma4DSparkConfig for from_pretrained).

License

• Weights and tokenizer: Apache 2.0, consistent with google/gemma-4-26B-A4B-it
• Vendored Python modules: DeepSpec / MIT — see NOTICE

Citation

@misc{cheng2026dsparkconfidencescheduledspeculativedecoding,
      title={DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation},
      author={Xin Cheng and Xingkai Yu and Chenze Shao and Jiashi Li and Yunfan Xiong and Yi Qian and Jiaqi Zhu and Shirong Ma and Xiaokang Zhang and Jiasheng Ye and Qinyu Chen and Chengqi Deng and Jiping Yu and Damai Dai and Zhengyan Zhang and Yixuan Wei and Yixuan Tan and Wenkai Yang and Runxin Xu and Yu Wu and Zhean Xu and Xuanyu Wang and Muyang Chen and Rui Tian and Xiao Bi and Zhewen Hao and Shaoyuan Chen and Huanqi Cao and Wentao Zhang and Anyi Xu and Huishuai Zhang and Dongyan Zhao and Wenfeng Liang},
      year={2026},
      eprint={2607.05147},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.05147},
}