Qwen3.8-27B-MLA-EXL3-4.0bpw

EXL3 4.0 bpw build of Qwen3.8-27B-MLA, the training-free Multi-head Latent Attention (MLA) retrofit of Qwen3.8-27B: 17.2 GB instead of 55.7 GB, with a calibrated FP8 KV cache by default. The smallest build of the family.

Serving: this checkpoint uses a custom attention architecture. It runs in vLLM with the qwen-mla-vllm plugin (experimental) plus the cuda-exl3 EXL3 runtime. It does not load in transformers or stock vLLM. Includes the base model's vision tower and MTP head.

Serving (vLLM plugin, experimental)

pip install "git+https://github.com/sootaugur/qwen-mla-vllm"      # installs vllm==0.27.1
pip install --no-build-isolation "git+https://github.com/Zeuss5/cuda-exl3"   # EXL3 runtime (compiles CUDA; needs nvcc)
vllm serve TelperionAI/Qwen3.8-27B-MLA-EXL3-4.0bpw --reasoning-parser qwen3

Images work as with the base model; for text-only serving add --limit-mm-per-prompt '{"image": 0, "video": 0}'. The KV cache is FP8 by default (calibrated per layer on this model); --kv-cache-dtype bfloat16 switches to bf16.

24 GB GPUs: serve text-only with the default FP8 KV cache and no MTP, --limit-mm-per-prompt '{"image": 0, "video": 0}' --max-model-len 32768: about 106k tokens of KV (measured under the memory budget vLLM uses on a 24 GB card). Vision and MTP need more memory than that leaves.

Speed (TP=1, default FP8 KV, no MTP): 69 / 301 / 1,056 tok/s at 1 / 8 / 32 concurrent requests, against 25 / 136 for the bf16 parent with FP8 KV at 1 / 8.

MTP speculative decoding (optional)*. The MTP head is included; enable it with --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'. *On an RTX PRO 6000 (TP=1, FP8 KV) it did not speed this build up: 69 / 301 tok/s at 1 / 8 concurrent requests without MTP, 40 / 236 with k=1, 54 / 309 with k=3. The quantized weights are already cheap to read there, and each verify step has a fixed cost on the FP8-cache path. On GPUs with less memory bandwidth weight reads dominate, so MTP is likely to help; measure it on your hardware.

The plugin is experimental: verified on RTX PRO 6000 (Blackwell) at TP=1 and TP=2 with vLLM 0.27.1, with little real-world use so far. FP8-cache decode runs on the plugin's Triton kernels.

What this is

The model is Qwen3.8-27B-MLA: see its card for the method, calibration data and results. Only the storage format changes here.

• Format: EXL3 (ExLlamaV3 trellis quantization), 4.0 bits per weight: MLPs, linear-attention and attention projections at 4 bits, lm_head at 6 bits, the MTP head at 4 bits. Runs in vLLM through cuda-exl3, which must be installed next to qwen-mla-vllm.
• Kept in bf16: the latent projections (kv_a_proj, k_up_proj, v_up_proj, k_rope_proj, and the kv_b_proj the plugin builds from them at load time), norms, embeddings and the vision tower.
• KV cache: FP8 by default, with per-layer scales calibrated on this model (absmax over a 3.7M-token corpus); config mla_kv_cache_scales.

How it was built. The retrofit differs from Qwen3.8-27B only in its 16 full-attention layers, so it was grafted onto TelperionAI/Qwen3.8-27B-EXL3-4.0bpw-v2, a quantized build of the base model:

• every quantized tensor of the MLPs, linear attention, o_proj (byte-identical in the retrofit) and the MTP head is reused unchanged;
• q_proj is reused unpermuted: the retrofit's q_proj is exactly the base q_proj with a fixed rope permutation of its query rows (verified bit-exact), so the plugin applies that permutation to the query at runtime (config mla_q_proj_base_layout) -- exact, and cheap;
• the latent projections and attention norms are the retrofit's bf16 tensors.

The reused tensors were calibrated on the base model's activations rather than the retrofit's.

The family

Two retrofits of Qwen3.8-27B, each in four weight formats. All keep the vision tower and MTP head, and all ship calibrated FP8 KV-cache scales.

Fidelity to the bf16 base model, teacher-forced on 562k held-out positions: top-1 disagreement where the base model is confident (top-1 vs top-2 logprob margin 2-5; lower is closer). Quantized builds measured with their default FP8 KV cache.

FP8 KV on the bf16 retrofits measured identical (1.09% / 2.47%); its effect is within run-to-run noise.

Capacity, full 262k-token sequences that fit at once (RTX PRO 6000, 96 GB, default settings with vision on, no MTP). Base model, bf16: 1.9 on one GPU.

Limitations

• Serving requires the experimental vLLM plugin (vLLM 0.27.1 only, for now) and cuda-exl3.
• Not yet benchmarked on downstream tasks (SWE-bench, HELMET); the fidelity measurement above is the evidence so far. A rebench of the quantized family is planned.
• Results are from one harness. Absolute scores differ across harnesses.

License and attribution

Released under Apache-2.0, the license of the base model. Derived from Qwen/Qwen3.8-27B by the Qwen team. Runs on cuda-exl3 (https://github.com/Zeuss5/cuda-exl3), an EXL3 runtime for vLLM; EXL3 is the ExLlamaV3 format by turboderp. This is an independent project, not affiliated with or endorsed by the Qwen team.