Qwen3.8-27B-MLA

Qwen3.8-27B with its attention KV cache compressed 2× by retrofitting Multi-head Latent Attention (MLA), without any training. In paired runs on SWE-bench Verified it resolved 0.421 against the original model's 0.395, while generating 12% fewer tokens.

Serving: this checkpoint uses a custom attention architecture. It runs in vLLM with the qwen-mla-vllm plugin (experimental; see Serving). It does not load in transformers or stock vLLM. Includes the base model's vision tower and MTP head. FP8 build: Qwen3.8-27B-MLA-FP8.

Serving (vLLM plugin, experimental)

This checkpoint is served by qwen-mla-vllm, a vLLM plugin. Install it and vLLM finds the architecture automatically. No fork or extra flags needed.

pip install "git+https://github.com/sootaugur/qwen-mla-vllm"      # installs vllm==0.27.1
vllm serve TelperionAI/Qwen3.8-27B-MLA --reasoning-parser qwen3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

--speculative-config turns on MTP speculative decoding (optional). Images work as with the base model; for text-only serving add --limit-mm-per-prompt '{"image": 0, "video": 0}'.

The plugin is experimental. This version has been verified on RTX PRO 6000 (Blackwell) at TP=1 and TP=2, with vLLM 0.27.1 only (earlier versions also ran on H100), and has seen little real-world use so far. The first start JIT-compiles the fast decode kernel, which needs nvcc and ninja. Without them the plugin falls back to Triton kernels: correct, but slower. See the plugin README for requirements, configuration and verification.

What this is

Qwen3.8-27B is a hybrid model: 48 Gated DeltaNet (linear attention) layers and 16 full-attention layers with grouped-query attention (24 query heads, 4 KV heads, head dim 256). Only the 16 full-attention layers keep a KV cache that grows with context, and those are what this release changes.

In each full-attention layer, the separate K and V projections are replaced by a shared low-rank latent that all 24 heads read, plus a small decoupled RoPE key. At inference the cache stores the latent instead of per-head keys and values:

• latent rank per layer: 256 / 768 / 1792 (mean 768), allocated by sensitivity;
• RoPE: 58 of 64 rotary dimensions kept per KV head. The three lowest-frequency rotary pairs turn only 3.4°, 2.1° and 1.2° across 131k tokens, so they carry content rather than position. They are not discarded: they stop rotating and move into the latent. This makes each cache row exactly 512 / 1024 / 2048 values with no memory-page padding (2.000× served instead of 1.882×), at a measured cost of 0.49% of attention logits at 131k tokens;
• cache row per layer: 512 / 1024 / 2048 values (vs 2048 for the original), 2.0× smaller on average.

The projections are fit training-free: whitened SVD of the original K/V weights under the covariance of each layer's inputs, measured on a calibration corpus. Nothing else in the model changes. Embeddings, MLPs, all 48 linear-attention layers, and the query projections (up to a fixed permutation) are byte-identical to the base model, as are the vision tower and the MTP head.

Files. Full bf16 weights (the complete model, not a delta, including the base model's vision tower and MTP head), tokenizer, chat template, preprocessor configs and the base model's generation_config.json. config.json declares Qwen3_5MLAForConditionalGeneration, the latent-cache architecture the vLLM plugin serves; the mla_* fields record the per-layer ranks and RoPE layout.

Calibration data (this matters: it is the only thing the fit sees):

• long-context QA: HotpotQA train passages (~62k tokens each) with the base model's own answers;
• agentic coding traces: community sessions from Qwen3.8-27B;
• software engineering: the base model's own reasoning and patches on SWE-bench train (22 repositories, none shared with SWE-bench Verified).

Moving the calibration to the model's own outputs (on-policy QA and SWE-bench train traces, replacing plain retrieval passages) cut the student's excess NLL on SWE-bench conversations by a third.

Is SWE-bench data in calibration a contamination risk? No training happens: no gradients and no labels. Calibration measures one statistic, the covariance of each attention layer's inputs, and the latent keeps the directions with the most variance under it. The SWE-bench traces come from the train split, and their 22 repositories are disjoint from the 12 in SWE-bench Verified. No Verified task, issue, or repository is seen. What calibration can do is specialise the latent toward a domain's activations, which is the intended effect.

Results

All comparisons are paired: same prompts, same harness, same serving settings, run on the same machine as the base model.

SWE-bench Verified

evalscope, BM25-retrieved context (princeton-nlp/SWE-bench_bm25_40K), thinking on, temperature 1.0, 500 instances, repeated runs.

The difference comes from fewer patches that fail to apply and fewer missing patches; the harness error rate is identical. Results cover SWE-bench Verified and HELMET (64k) only; we make no claims about other domains.

HELMET long-context (64k tokens)

KILT RAG (NQ, TriviaQA, HotpotQA, PopQA) and JSON key-value retrieval, 100 questions per task, chat template, thinking off.

Within noise of the base model at n=100 per task.

Fidelity to the base model

Teacher-forced NLL on text the base model generated, excess over the base model (lower is closer):

Generation behaviour: none of 1,500 SWE-bench generations hit the length limit; 1 had more than 20% repeated text (base model: 0).

KV cache and tensor parallelism

MLA's latent is shared by every head, so under tensor parallelism it is replicated on each GPU, while the base model's 4 KV heads are split across GPUs. Per-GPU KV per token:

(The base model's 4 KV heads cannot be split beyond TP=4, so its per-GPU cost stops falling.)

This model is for single-GPU (TP=1) serving, where it doubles how many long-context requests fit in memory (measured: 1.75× served KV capacity in vLLM at TP=1, including the linear-attention state). For TP=2 see Qwen3.8-27B-GLA-g2, which splits its latent across GPUs.

Speed

RTX PRO 6000 (Blackwell), bf16 weights and KV, vLLM 0.27.1 with the plugin, 16k-token prompts, 256 decode steps. "Max" is each model's own largest batch that fits in KV memory.

Single-stream latency is within 5% of the base model, and the larger batch gives 1.34× the peak throughput. Prefill matches the base model to within 3%. At TP=2 the replicated latent leaves this model slightly behind the base model (82 vs 92 sequences, 988 vs 1,043 tok/s). Use GLA-g2 there.

Speculative decoding (MTP)*

The base model's MTP head is included unchanged, and it drafts for the retrofit as well as for the base model: mean accepted tokens per step at k=3 are 1.86 vs 1.82 (retrofit vs base). Decode throughput, tokens/s at 1 / 8 / 32 concurrent requests (64 mixed prompts, thinking on, temperature 0.6, top-p 0.95, top-k 20, bf16, RTX PRO 6000):

The retrofit's speedup is smaller because a verify step currently reads the latent cache once per draft token in the plugin; a single causal pass per request is planned.

*MTP measurements are from one GPU type (RTX PRO 6000, very high memory bandwidth). The benefit depends on hardware: where weight reads dominate decode (lower-bandwidth GPUs) it is likely larger. With an FP8 KV cache on that GPU, k=3 gave 1.6x (bf16 build) and 1.3x (FP8 build) at 1 concurrent request; measure on your own hardware.

Vision

The vision tower is the base model's, unchanged. Paired against the base model on the same items, greedy, thinking off (300 ChartQA test questions, relaxed accuracy; 300 DocVQA validation questions, ANLS):

Neither difference is significant; 90–95% of answers are identical to the base model's.

FP8 KV cache

Calibrated per-layer FP8 KV scales ship in config.json (absmax over a 3.7M-token corpus, measured in the served model). Serve with --kv-cache-dtype fp8 for twice the KV capacity: on one GPU at 262k-token requests the MLA model then holds 7.3 sequences, 3.8x the bf16 base model. In teacher-forced tests the FP8 cache is indistinguishable from run-to-run noise. FP8-cache decode runs on Triton kernels, so it is about 15-25% slower than the default bf16 path.

Roadmap

• vLLM plugin: qwen-mla-vllm (experimental)
• Optimized decode kernels (patched FlashInfer MLA, two-pass Triton for the widest latent)
• FP8: Qwen3.8-27B-MLA-FP8
• Vision tower and MTP head (speculative decoding) included
• 4-bit variants: INT4 (AWQ+GPTQ) and EXL3 4.0 bpw
• Calibrated FP8 KV cache
• Method and code release: calibration corpus builders, covariance tooling, the exporter, and evaluation harness

The family

Two retrofits of Qwen3.8-27B, each in four weight formats. All keep the vision tower and MTP head, and all ship calibrated FP8 KV-cache scales.

Fidelity to the bf16 base model, teacher-forced on 562k held-out positions: top-1 disagreement where the base model is confident (top-1 vs top-2 logprob margin 2-5; lower is closer). Quantized builds measured with their default FP8 KV cache.

FP8 KV on the bf16 retrofits measured identical (1.09% / 2.47%); its effect is within run-to-run noise.

Capacity, full 262k-token sequences that fit at once (RTX PRO 6000, 96 GB, default settings with vision on, no MTP). Base model, bf16: 1.9 on one GPU.

Limitations

• Serving requires the experimental vLLM plugin (vLLM 0.27.1 only, for now).
• Evaluated on SWE-bench Verified and HELMET (64k); other domains and much longer contexts are untested.
• Results are from one harness and sampling configuration; absolute scores differ across harnesses. The paired differences are the meaningful numbers.
• The compression only helps at TP=1 (see above).

License and attribution

Released under Apache-2.0, the license of the base model. Derived from Qwen/Qwen3.8-27B by the Qwen team. This is an independent project, not affiliated with or endorsed by the Qwen team.

Citation

@misc{telperion2026qwen38mla,
  title  = {Qwen3.8-27B-MLA: training-free Multi-head Latent Attention retrofit},
  author = {TelperionAI},
  year   = {2026},
  url    = {https://huggingface.co/TelperionAI/Qwen3.8-27B-MLA}
}