Qwen3.8-27B-MLA-FP8

FP8 build of Qwen3.8-27B-MLA, the training-free Multi-head Latent Attention (MLA) retrofit of Qwen3.8-27B: 31.5 GB instead of 55.7 GB, with the same 2× smaller KV cache. On SWE-bench Verified it resolved 0.403 over 3 runs, against 0.395 for the bf16 base model and 0.399 for Qwen's own FP8 build.

Serving: this checkpoint uses a custom attention architecture. It runs in vLLM with the qwen-mla-vllm plugin (experimental; see Serving). It does not load in transformers or stock vLLM. Includes the base model's vision tower and MTP head.

All four include the vision tower and the MTP head.

Serving (vLLM plugin, experimental)

pip install "git+https://github.com/sootaugur/qwen-mla-vllm"      # installs vllm==0.27.1
vllm serve TelperionAI/Qwen3.8-27B-MLA-FP8 --reasoning-parser qwen3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

--speculative-config turns on MTP speculative decoding (optional*). Images work as with the base model; for text-only serving add --limit-mm-per-prompt '{"image": 0, "video": 0}'.

The plugin is experimental. This version has been verified on RTX PRO 6000 (Blackwell) at TP=1 and TP=2, with vLLM 0.27.1 only, and has seen little real-world use so far. It needs a GPU that vLLM runs block-FP8 on (Hopper or Blackwell; only Blackwell has been verified). There vLLM selects CutlassFp8BlockScaledMMKernel. The first start JIT-compiles the fast decode kernel, which needs nvcc and ninja. Without them the plugin falls back to Triton kernels: correct, but slower. See the plugin README for requirements, configuration and verification.

Speed (MLA-FP8, TP=1, RTX PRO 6000, default FP8 KV): 43 / 225 tok/s at 1 / 8 concurrent requests without MTP, 54 / 294 with MTP k=3*. A full benchmark has not been run yet. For bf16 numbers, see the parent model.

*MTP measurements are from one GPU type (RTX PRO 6000, very high memory bandwidth). The benefit depends on hardware: where weight reads dominate decode (lower-bandwidth GPUs) it is likely larger. With an FP8 KV cache on that GPU, k=3 gave 1.6x (bf16 build) and 1.3x (FP8 build) at 1 concurrent request; measure on your own hardware.

What this is

The model is Qwen3.8-27B-MLA: see its card for the method, calibration data and full results. Only the storage format of the weights changes here.

• Format: compressed-tensors. Weights are FP8 e4m3 with 128×128 block scales. Activations are quantized to FP8 dynamically per token, in groups of 128.
• Kept in bf16: the latent projections (kv_a_proj, k_up_proj, v_up_proj, k_rope_proj, and the kv_b_proj the plugin builds from them at load time), the linear-attention in_proj_a / in_proj_b and norms, the embeddings and lm_head, the vision tower and the MTP head (both the base model's, unchanged).
• KV cache: FP8 by default, with per-layer scales calibrated on this model (config mla_kv_cache_scales); --kv-cache-dtype bfloat16 switches to bf16. The SWE-bench results below were measured with a bf16 KV cache; FP8 KV measured within run-to-run noise (see The family).

How it was built. The retrofit differs from Qwen3.8-27B only in its 16 full-attention layers, so it was grafted onto TelperionAI/Qwen3.8-27B-FP8-block-AWQ, an FP8 block build of the base model with AWQ smoothing of the MLPs:

• MLPs, linear-attention layers, o_proj, embeddings and norms: taken unchanged from that build. The AWQ smoothing touches only the MLP mappings, not the attention inputs, so no attention tensor needed rescaling.
• q_proj (a fixed permutation of the base q_proj under partial RoPE): re-quantized from the retrofit's bf16 weights with the recipe's own calibration-free FP8 step (block 128×128, amax/448). Applied to the base weights, that step reproduces the published FP8 q_proj / k_proj / o_proj bit for bit.
• Latent projections and attention norms: the retrofit's bf16 tensors.

Results

All comparisons are paired: same prompts, same harness, same serving settings, against the same bf16 base-model runs as the parent card.

SWE-bench Verified

evalscope, BM25-retrieved context (princeton-nlp/SWE-bench_bm25_40K), thinking on, temperature 1.0, 500 instances.

Against its own bf16 parent (paired, 3 runs each), the FP8 build differs by −1.7 [−3.8, +0.3] points, which is not significant. Results cover SWE-bench Verified only. HELMET was not re-run on the FP8 build. Vision spot check (100 ChartQA / 100 DocVQA items, with MTP on): 91.0 / 96.8; see the parent's card for the paired vision comparison.

Fidelity to the base model

Teacher-forced NLL on text the bf16 base model generated, excess over the bf16 base (lower is closer):

FP8 adds about the same error to the retrofit (+0.004 to +0.006) as it adds to the base model. The two sources of error simply add.

Generation check: in thinking mode on SWE-bench prompts, all generations stopped normally, patches were present, and lengths were within sampling variation of the base model.

KV cache and tensor parallelism

Identical to the parent; see its card. This model is meant for single-GPU (TP=1) serving.

The family

Two retrofits of Qwen3.8-27B, each in four weight formats. All keep the vision tower and MTP head, and all ship calibrated FP8 KV-cache scales.

Fidelity to the bf16 base model, teacher-forced on 562k held-out positions: top-1 disagreement where the base model is confident (top-1 vs top-2 logprob margin 2-5; lower is closer). Quantized builds measured with their default FP8 KV cache.

FP8 KV on the bf16 retrofits measured identical (1.09% / 2.47%); its effect is within run-to-run noise.

Capacity, full 262k-token sequences that fit at once (RTX PRO 6000, 96 GB, default settings with vision on, no MTP). Base model, bf16: 1.9 on one GPU.

Limitations

• Serving requires the experimental vLLM plugin (vLLM 0.27.1 only, for now).
• Only a spot check of speed for this build (see Serving).
• Evaluated on SWE-bench Verified and teacher-forced fidelity. Other domains and long-context benchmarks were not re-run for the FP8 build.
• Results are from one harness and sampling configuration. Absolute scores differ across harnesses, so the paired differences are the meaningful numbers.

License and attribution

Released under Apache-2.0, the license of the base model. Derived from Qwen/Qwen3.8-27B by the Qwen team. This is an independent project, not affiliated with or endorsed by the Qwen team.

Citation

@misc{telperion2026qwen38mlafp8,
  title  = {Qwen3.8-27B-MLA-FP8: FP8 build of a training-free latent attention retrofit},
  author = {TelperionAI},
  year   = {2026},
  url    = {https://huggingface.co/TelperionAI/Qwen3.8-27B-MLA-FP8}
}