Uno adapter for Gemma 4 26B A4B (38.6k open prompts, trained at K=5)

A rank-16 Uno drafter for Gemma 4 26B A4B, the model BroadNet runs in production in English and Arabic. With Uno for vLLM v0.4.4 and its recommended settings, on BroadNet's real production traffic on one RTX 3090, Gemma 4 serves 721 tokens per second with eight requests in flight and 523 with four (stock DFlash K8 in vLLM 0.30.0: 679 and 497), and one request at a time it decodes 1.69x faster than plain decoding and takes 7.6 % less time per token than stock DFlash K8 in vLLM 0.30.0, with the same output distribution as the model on its own. On long documents the lead grows: 1.37x plain at 14k-token prompts and 1.31x at 28k; at 28k, stock DFlash K8 in vLLM 0.30.0 is slower than plain decoding. Trained on 38,560 open prompts on one 24 GB RTX 3090; its last training stage drafts five tokens per block, the number the recommended settings serve. The adapter released with v0.4.3 stays at revision s5, the 28.6k-prompt adapter at s3 and the 10k-prompt adapter at p10k.

226 tokens per second per request on one RTX 3090 at 32k context, one request at a time on that traffic (stock DFlash K8 in vLLM 0.30.0: 209, plain decoding: 134).

Links. Release: Uno for vLLM v0.4.4 (serving code, container ghcr.io/brntech/vllm-uno:0.4.4, validation and evidence). Method: Uno by IFM and collaborators.

What it is

What it measures

One RTX 3090, the v0.4.4 container, the recommended settings (UNO_RECOMMENDED=1): K=5 Uno drafts, prompt lookup on (UNO_PLOOKUP_L=2, UNO_PLOOKUP_MAX_CTX=4096), draft passes routed through 4 of the 8 experts per token (UNO_DRAFT_MOE_TOPK=4), split-KV attention for steps with several requests (UNO_GEMMA_SPLITKV_MULTI=1) with CUDA graphs for up to 8 requests, the shipped 64k draft vocabulary. Plain decoding and the DFlash drafter (z-lab, K=8, run on the stock vllm/vllm-openai:v0.30.0 image with the same serve flags) were measured in the same session, three rounds each with the arm order rotated, every server started fresh for its workload and sent two warmup requests (the first two production requests) before the timed ones. Production traffic = 72 real requests. With several requests in flight each figure is the median of the three rounds, with the range in brackets; one request at a time the table gives each round, and tokens per second at the median of the three rounds; for long prompts each figure is the median of all 24 requests of the three rounds pooled. Speed-ups over plain are ratios of these medians. This adapter's figures are rounded so that none is better than measured.

Several requests at a time (the 72 requests with four or eight in flight, wall clock over the whole batch):

This adapter took 5.8 %, 5.8 % and 7.0 % less time per token than stock DFlash K8 in vLLM 0.30.0 at eight in flight (rounds 1, 2 and 3), and 6.7 %, 6.0 % and 3.7 % less at four.

One request at a time (HTTP clock: completion tokens over request time):

This adapter took 4.0 %, 7.8 % and 8.3 % less time per token than stock DFlash K8 in vLLM 0.30.0 (rounds 1, 2 and 3).

Prompt lookup. Since v0.4.3 there is a second source of candidates: after Uno's five drafted tokens, two more are copied from the request itself (the tokens that followed the last place the current text appeared in the prompt or the answer so far). In a separate session with an earlier training stage (K=4; v0.4.3 release evidence), lookup with the length gate below cut the time per token by 2.9 % on this traffic against lookup off. On long prompts lookup did not pay: without the gate it was 4.7 % and 4.3 % slower than lookup off at 14k and 28k tokens. So v0.4.3 switches it off per request once a request's context passes 4,096 tokens: long requests then verify only Uno's drafts, as in v0.4.2.

Long prompts. Open-data documents at three lengths (4 document sets x 2 tasks, a summary and an original story, up to 384 output tokens each, one at a time), same session and settings, on the one-at-a-time servers after the production requests; median decode time per output token, all three rounds pooled (24 requests per cell):

Against stock DFlash K8 in vLLM 0.30.0, this adapter took 34.8 %, 43.8 % and 38.0 % less time per token at 14k (rounds 1, 2 and 3) and 48.1 %, 45.5 % and 47.5 % less at 28k. Prompts above 4,096 tokens verify only Uno's drafts (the length gate).

Lossless. Every candidate, drafted by Uno or copied from the prompt, is checked by vLLM's own rejection sampler against the full model before it is emitted, so the output distribution is the model's own; in the Uno authors' words, Uno "accelerates generation without sacrificing the quality of the underlying AR model" (Sahoo et al.). A copied token is a fixed proposal, which the sampler accepts with exactly the model's probability for it. The release's CPU tests run vLLM's real rejection kernels over the served shape (five drafts plus two lookup tokens) and check the emitted tokens against the model's distribution under greedy decoding, temperature 1, temperature 0.7 with top-p 0.9, vocabularies above 8,192 tokens, and batches that mix short requests (with lookup) and long ones (without). The adapter, and routing the draft pass through 4 experts, change only which tokens Uno proposes: the model's verification pass always uses all 8 experts per token, and the sampler checks each proposal against the draft distribution it was actually drawn from. With several requests in flight, v0.4.4 computes their attention in one split-KV launch; a GPU test on the release image checks that each request's attention output is bit for bit the one it gets when it runs alone.

Why we built it

BroadNet serves Gemma 4 26B A4B to its clients from its own GPUs, for agent workloads in English and Arabic: long system prompts, tool calls, structured JSON, multi-step coordination. Faster decoding on the same card means faster answers and more work per GPU, with the model's output distribution unchanged. More open prompts kept making the drafter better, and copying from the prompt adds what a drafter cannot know in advance: the exact strings the request itself contains. This adapter's last training stage drafts as many tokens per block as it is asked to draft when served.

What it is not

• Not a quality upgrade: the drafter only proposes tokens; the model decides every one.
• The recommended settings are off unless you turn them on (UNO_RECOMMENDED=1, below). Without them this adapter runs on the v0.4.2 path (K=4, no lookup, 8-expert drafts); its speed there was not measured for this card.
• The 4-expert draft setting (UNO_DRAFT_MOE_TOPK=4, part of the recommended settings) runs only on RTX 30-series class GPUs (compute capability 8.6) and refuses to start elsewhere; there, add UNO_DRAFT_MOE_TOPK= (empty). It runs only inside captured CUDA graphs, so with it the server refuses to start when no draft graph will be captured (--enforce-eager, or a CUDA graph mode without full decode graphs), and checks at startup that every draft batch it can be asked for has a captured graph: with the recommended settings (up to 8 requests, capture sizes up to 64), --max-num-seqs above 8 stops at startup with a message naming the missing batch sizes. Only UNO_K=5 with lookup UNO_PLOOKUP_L=2 was measured; with the multi-request path on, the server refuses more than 9 verify rows per request (UNO_K + UNO_PLOOKUP_L + 1).
• Measured on one GPU type (RTX 3090). Other checkpoints, other Gemma 4 sizes and other GPUs need their own validation.
• All 72 production requests are in English; speed on Arabic traffic is not measured.

How to run it

With the v0.4.4 container, the model cache and this adapter mounted, and the gemma4 profile selected (the profile pins the checkpoint revision above, serves at 32k context on vLLM's hybrid KV cache manager with split-KV draft attention and the 64k draft vocabulary on), plus the recommended settings in one switch:

docker run --rm --name vllm-uno-gemma --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v /path/to/hf-cache:/root/.cache/huggingface \
  -v /path/to/this-adapter:/adapter:ro \
  -e UNO_PROFILE=gemma4 \
  -e UNO_RECOMMENDED=1 \
  ghcr.io/brntech/vllm-uno:0.4.4 \
  cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit /adapter \
  -- --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4

The flags after -- turn on Gemma 4's tool-call and reasoning parsers, as on the measured servers. UNO_RECOMMENDED=1 sets UNO_K=5, UNO_PLOOKUP_L=2, UNO_PLOOKUP_MAX_CTX=4096, UNO_DRAFT_MOE_TOPK=4 and UNO_GEMMA_SPLITKV_MULTI=1 (with capture sizes up to 64); a variable you set yourself wins. Set to empty, UNO_PLOOKUP_L= turns prompt lookup off, UNO_DRAFT_MOE_TOPK= drafts with all 8 experts and UNO_GEMMA_SPLITKV_MULTI= turns the multi-request path off; an empty UNO_K=, or an empty UNO_PLOOKUP_MAX_CTX= while lookup is on, is refused at startup. The server prints the settings it used at startup.

The served model name is uno-gemma4-26b-a4b. With the recommended settings the KV cache holds 75,731 to 75,768 tokens at 32k context on a 24 GB RTX 3090. The earlier adapters stay available: hf download Broadnet/gemma-4-26B-A4B-uno-adapter --revision s5 --local-dir /path/to/s5-adapter (38.6k prompts, trained with four-row draft blocks, v0.4.3), --revision s3 (28.6k prompts, v0.4.2) or --revision p10k (10k prompts, v0.4.0 and v0.4.1); mount that directory as /adapter.

Licence and where the training data comes from

The adapter weights are Apache 2.0, like the Gemma 4 base model. They were trained on 38,560 prompts drawn from public datasets; the answers the drafter learned from were written by the Gemma 4 model itself. No private or production data was used for training. The 10,000 rows added since the 28.6k-prompt adapter come from the same datasets and subsets; the seventh stage adds no new rows. Dataset licences as published on each dataset card (revisions pinned in the corpus recipe):

The 64k draft vocabulary shipped with v0.4.0 is ranked on the first 10,000 of these prompts and the model's answers to them.

Credits

Uno is the work of Subham Sekhar Sahoo, Lingjie Chen, Khiem Pham, Jonathan Geuter, Chaitanya Dwivedi, Varad Pimpalkhute, Yash Akhauri, Alexander Moreno, Mikhail Yurochkin, Zhenting Wang, Mostafa Elhoushi, Nolan Dey, Shane Bergsma, Joel Hestness, John Thickstun, Eric Xing and Zhengzhong Liu: "Unlocking Lossless Speedups in LLMs via Discrete Diffusion" (arXiv:2609.04010; code at ifm-ai/uno). Gemma 4 is by Google; the AWQ checkpoint is by cyankiwi; the DFlash comparator drafter is by z-lab; serving runs on vLLM. This adapter, the vLLM integration and every measurement on this card are BroadNet's independent implementation, by the BroadNet Research Team.