Qwen3.8-27B-MLA-INT4

INT4 (AWQ + GPTQ) build of Qwen3.8-27B-MLA, the training-free Multi-head Latent Attention (MLA) retrofit of Qwen3.8-27B: 22.9 GB instead of 55.7 GB, with a calibrated FP8 KV cache by default. The most faithful 4-bit build of the family.

Serving: this checkpoint uses a custom attention architecture. It runs in vLLM with the qwen-mla-vllm plugin (experimental). It does not load in transformers or stock vLLM. Includes the base model's vision tower and MTP head.

Serving (vLLM plugin, experimental)

pip install "git+https://github.com/sootaugur/qwen-mla-vllm"      # installs vllm==0.27.1
vllm serve TelperionAI/Qwen3.8-27B-MLA-INT4 --reasoning-parser qwen3

Images work as with the base model; for text-only serving add --limit-mm-per-prompt '{"image": 0, "video": 0}'. The KV cache is FP8 by default (calibrated per layer on this model); --kv-cache-dtype bfloat16 switches to bf16.

32 GB GPUs: serve text-only with the default FP8 KV cache, --limit-mm-per-prompt '{"image": 0, "video": 0}' --max-model-len 32768: about 250k tokens of KV (measured under the memory budget vLLM uses on a 32 GB card).

Speed (TP=1, default FP8 KV, no MTP): 59 / 279 / 963 tok/s at 1 / 8 / 32 concurrent requests, against 25 / 136 for the bf16 parent with FP8 KV at 1 / 8.

MTP speculative decoding (optional)*. The MTP head is included; enable it with --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'. *On an RTX PRO 6000 (TP=1, FP8 KV) it did not speed this build up: 59 / 279 tok/s at 1 / 8 concurrent requests without MTP, 42 / 258 with k=1, 56 / 317 with k=3. The quantized weights are already cheap to read there, and each verify step has a fixed cost on the FP8-cache path. On GPUs with less memory bandwidth weight reads dominate, so MTP is likely to help; measure it on your hardware.

The plugin is experimental: verified on RTX PRO 6000 (Blackwell) at TP=1 and TP=2 with vLLM 0.27.1, with little real-world use so far. FP8-cache decode runs on the plugin's Triton kernels.

What this is

The model is Qwen3.8-27B-MLA: see its card for the method, calibration data and results. Only the storage format changes here.

• Format: compressed-tensors. MLPs of layers 0-55 and the linear-attention projections: INT4, group 32, asymmetric (AWQ smoothing + GPTQ). Attention q_proj / o_proj and the MLPs of layers 56-63: INT8, group 128, symmetric. Weight-only; activations stay bf16.
• Kept in bf16: the latent projections (kv_a_proj, k_up_proj, v_up_proj, k_rope_proj, and the kv_b_proj the plugin builds from them at load time), the linear-attention in_proj_a / in_proj_b and norms, embeddings, lm_head, the vision tower and the MTP head.
• KV cache: FP8 by default, with per-layer scales calibrated on this model (absmax over a 3.7M-token corpus); config mla_kv_cache_scales.

How it was built. The retrofit differs from Qwen3.8-27B only in its 16 full-attention layers, so it was grafted onto TelperionAI/Qwen3.8-27B-INT4-AWQ-GPTQ-gdn4, a quantized build of the base model:

• every quantized tensor of the MLPs, linear attention, o_proj (byte-identical in the retrofit) and the MTP head is reused unchanged;
• q_proj is reused unpermuted: the retrofit's q_proj is exactly the base q_proj with a fixed rope permutation of its query rows (verified bit-exact), so the plugin applies that permutation to the query at runtime (config mla_q_proj_base_layout) -- exact, and cheap;
• the latent projections and attention norms are the retrofit's bf16 tensors.

The reused tensors were calibrated on the base model's activations rather than the retrofit's.

The family

Two retrofits of Qwen3.8-27B, each in four weight formats. All keep the vision tower and MTP head, and all ship calibrated FP8 KV-cache scales.

Fidelity to the bf16 base model, teacher-forced on 562k held-out positions: top-1 disagreement where the base model is confident (top-1 vs top-2 logprob margin 2-5; lower is closer). Quantized builds measured with their default FP8 KV cache.

FP8 KV on the bf16 retrofits measured identical (1.09% / 2.47%); its effect is within run-to-run noise.

Capacity, full 262k-token sequences that fit at once (RTX PRO 6000, 96 GB, default settings with vision on, no MTP). Base model, bf16: 1.9 on one GPU.

Limitations

• Serving requires the experimental vLLM plugin (vLLM 0.27.1 only, for now).
• Not yet benchmarked on downstream tasks (SWE-bench, HELMET); the fidelity measurement above is the evidence so far. A rebench of the quantized family is planned.
• Results are from one harness. Absolute scores differ across harnesses.

License and attribution

Released under Apache-2.0, the license of the base model. Derived from Qwen/Qwen3.8-27B by the Qwen team. This is an independent project, not affiliated with or endorsed by the Qwen team.