Qwen3.8-27B-GLA-g2

Qwen3.8-27B with its attention KV cache compressed 2×, including under 2-GPU tensor parallelism, by retrofitting Grouped Latent Attention (GLA) with 2 groups, without any training. In paired runs on SWE-bench Verified it resolved 0.394 against the original model's 0.395, while generating 14% fewer tokens.

Serving: this checkpoint uses a custom attention architecture. It runs in vLLM with the qwen-mla-vllm plugin (experimental; see Serving). It does not load in transformers or stock vLLM. Includes the base model's vision tower and MTP head. FP8 build: Qwen3.8-27B-GLA-g2-FP8.

Serving (vLLM plugin, experimental)

This checkpoint is served by qwen-mla-vllm, a vLLM plugin. Install it and vLLM finds the architecture automatically. No fork or extra flags needed.

pip install "git+https://github.com/sootaugur/qwen-mla-vllm"      # installs vllm==0.27.1
vllm serve TelperionAI/Qwen3.8-27B-GLA-g2 --tensor-parallel-size 2 --reasoning-parser qwen3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

--speculative-config turns on MTP speculative decoding (optional). Images work as with the base model; for text-only serving add --limit-mm-per-prompt '{"image": 0, "video": 0}'.

The plugin is experimental. This version has been verified on RTX PRO 6000 (Blackwell) at TP=1 and TP=2, with vLLM 0.27.1 only (earlier versions also ran on H100), and has seen little real-world use so far. The first start JIT-compiles the fast decode kernel, which needs nvcc and ninja. Without them the plugin falls back to Triton kernels: correct, but slower. See the plugin README for requirements, configuration and verification.

On PCIe-only GPUs (no NVLink, e.g. RTX PRO or GeForce), add --disable-custom-all-reduce at TP=2.

What this is

Qwen3.8-27B is a hybrid model: 48 Gated DeltaNet (linear attention) layers and 16 full-attention layers with grouped-query attention (24 query heads, 4 KV heads, head dim 256). Only the 16 full-attention layers keep a KV cache that grows with context, and those are what this release changes.

In each full-attention layer, the K and V projections are replaced by a low-rank latent split into 2 groups. Each group of 12 query heads reads only its own half of the latent, plus a small decoupled RoPE key. Because the groups are independent, the cache can be split across 2 GPUs just as the original KV heads were. Plain MLA (Qwen3.8-27B-MLA) shares one latent across all heads and must replicate it on every GPU.

• latent rank per layer: 256 / 768 / 1792 (mean 768), split evenly across the 2 groups;
• RoPE: 58 of 64 rotary dimensions kept per KV head. The three lowest-frequency rotary pairs turn only 3.4°, 2.1° and 1.2° across 131k tokens, so they carry content rather than position. They are not discarded: they stop rotating and move into the latent. This makes each cache row exactly 512 / 1024 / 2048 values with no memory-page padding (2.000× served instead of 1.882×), at a measured cost of 0.49% of attention logits at 131k tokens;
• cache row per layer: 512 / 1024 / 2048 values (vs 2048 for the original), 2.0× smaller on average.

The projections are fit training-free: whitened SVD of the original K/V weights under the covariance of each layer's inputs, measured on a calibration corpus. Nothing else in the model changes. Embeddings, MLPs, all 48 linear-attention layers, and the query projections (up to a fixed permutation) are byte-identical to the base model, as are the vision tower and the MTP head.

Files. Full bf16 weights (the complete model, not a delta, including the base model's vision tower and MTP head), tokenizer, chat template, preprocessor configs and the base model's generation_config.json. config.json declares Qwen3_5MLAForConditionalGeneration, the latent-cache architecture the vLLM plugin serves; the mla_* fields record the per-layer ranks and RoPE layout.

Calibration data (the only thing the fit sees):

• long-context QA: HotpotQA train passages (~62k tokens each) with the base model's own answers;
• agentic coding traces: community sessions from Qwen3.8-27B;
• software engineering: the base model's own reasoning and patches on SWE-bench train (22 repositories, none shared with SWE-bench Verified).

Adding the model's own SWE-bench train traces to calibration cut excess NLL on SWE-bench conversations by 31%. It also moved SWE-bench Verified from −1.1 to −0.1 points relative to the base model, compared with the same architecture calibrated without them.

Is SWE-bench data in calibration a contamination risk? No training happens: no gradients and no labels. Calibration measures one statistic, the covariance of each attention layer's inputs, and the latent keeps the directions with the most variance under it. The SWE-bench traces come from the train split, and their 22 repositories are disjoint from the 12 in SWE-bench Verified. No Verified task, issue, or repository is seen. What calibration can do is specialise the latent toward a domain's activations, which is the intended effect.

Results

All comparisons are paired: same prompts, same harness, same serving settings, run on the same machine as the base model.

SWE-bench Verified

evalscope, BM25-retrieved context (princeton-nlp/SWE-bench_bm25_40K), thinking on, temperature 1.0, 500 instances, repeated runs.

The two models agree on individual instances about as often as two runs of the base model agree with each other. Results cover SWE-bench Verified and HELMET (64k) only; we make no claims about other domains.

HELMET long-context (64k tokens)

KILT RAG (NQ, TriviaQA, HotpotQA, PopQA) and JSON key-value retrieval, 100 questions per task, chat template, thinking off.

Within noise of the base model at n=100 per task.

Fidelity to the base model

Teacher-forced NLL on text the base model generated, excess over the base model (lower is closer):

Generation behaviour: no increase in repetition or runaway generations relative to the base model (0 of 1,500 SWE-bench generations hit the length limit or looped).

KV cache and tensor parallelism

Per-GPU KV per token (the plugin stores each group's latent on its own GPU at TP=2):

This model is built for TP=1 and TP=2. It can be served at TP=4 or TP=8, but the 2 groups cannot be split further, so each group's latent is replicated and the advantage shrinks to about 1.14×. (The base model's 4 KV heads likewise stop splitting beyond TP=4.)

Speed

RTX PRO 6000 (Blackwell), bf16 weights and KV, vLLM 0.27.1 with the plugin, 16k-token prompts, 256 decode steps. "Max" is each model's own largest batch that fits in KV memory.

At TP=2 it holds 1.8× the sequences and gives 1.37× the peak throughput, and it is faster per step at equal batch. Single-stream latency is within 7% of the base model. Prefill matches the base model to within 3%. At TP=1 it serves like plain MLA (40 sequences, about 505 tok/s at max batch).

Speculative decoding (MTP)*

The base model's MTP head is included unchanged, and it drafts for the retrofit as well as for the base model: mean accepted tokens per step at k=3 are 1.83 vs 1.86 (retrofit vs base). Decode throughput, tokens/s at 1 / 8 / 32 concurrent requests (64 mixed prompts, thinking on, temperature 0.6, top-p 0.95, top-k 20, bf16, RTX PRO 6000):

The retrofit's speedup is smaller because a verify step currently reads the latent cache once per draft token in the plugin; a single causal pass per request is planned.

*MTP measurements are from one GPU type (RTX PRO 6000, very high memory bandwidth). The benefit depends on hardware: where weight reads dominate decode (lower-bandwidth GPUs) it is likely larger. With an FP8 KV cache on that GPU, k=3 gave 1.6x (bf16 build) and 1.3x (FP8 build) at 1 concurrent request; measure on your own hardware.

Vision

The vision tower is the base model's, unchanged. Paired against the base model on the same items, greedy, thinking off (300 ChartQA test questions, relaxed accuracy; 300 DocVQA validation questions, ANLS):

Neither difference is significant; 84–92% of answers are identical to the base model's.

FP8 KV cache

Calibrated per-layer FP8 KV scales ship in config.json (absmax over a 3.7M-token corpus, measured in the served model). Serve with --kv-cache-dtype fp8 for twice the KV capacity: on one GPU at 262k-token requests the MLA model then holds 7.3 sequences, 3.8x the bf16 base model. In teacher-forced tests the FP8 cache is indistinguishable from run-to-run noise. FP8-cache decode runs on Triton kernels, so it is about 15-25% slower than the default bf16 path.

Roadmap

• vLLM plugin: qwen-mla-vllm (experimental)
• Per-group cache layout and decode kernels (the TP=2 2× in serving)
• FP8: Qwen3.8-27B-GLA-g2-FP8
• Vision tower and MTP head (speculative decoding) included
• 4-bit variants: INT4 (AWQ+GPTQ) and EXL3 4.0 bpw
• Calibrated FP8 KV cache
• Method and code release: calibration corpus builders, covariance tooling, the exporter, and evaluation harness

The family

Two retrofits of Qwen3.8-27B, each in four weight formats. All keep the vision tower and MTP head, and all ship calibrated FP8 KV-cache scales.

Fidelity to the bf16 base model, teacher-forced on 562k held-out positions: top-1 disagreement where the base model is confident (top-1 vs top-2 logprob margin 2-5; lower is closer). Quantized builds measured with their default FP8 KV cache.

FP8 KV on the bf16 retrofits measured identical (1.09% / 2.47%); its effect is within run-to-run noise.

Capacity, full 262k-token sequences that fit at once (RTX PRO 6000, 96 GB, default settings with vision on, no MTP). Base model, bf16: 1.9 on one GPU.

Limitations

• Serving requires the experimental vLLM plugin (vLLM 0.27.1 only, for now).
• Evaluated on SWE-bench Verified and HELMET (64k); other domains and much longer contexts are untested.
• Results are from one harness and sampling configuration; absolute scores differ across harnesses. The paired differences are the meaningful numbers.
• Compression falls to about 1.14× beyond TP=2.

License and attribution

Released under Apache-2.0, the license of the base model. Derived from Qwen/Qwen3.8-27B by the Qwen team. This is an independent project, not affiliated with or endorsed by the Qwen team.

Citation

@misc{telperion2026qwen38gla,
  title  = {Qwen3.8-27B-GLA-g2: training-free Grouped Latent Attention retrofit},
  author = {TelperionAI},
  year   = {2026},
  url    = {https://huggingface.co/TelperionAI/Qwen3.8-27B-GLA-g2}
}