Qwen3.8-27B-MLA · EXL3 4.50 bpw

An EXL3 quant of TelperionAI/Qwen3.8-27B-MLA, sized for a single 24 GB GPU. On an RTX 3090 with the int8 KV cache it serves a 249k-token context with MTP speculative decoding and vision on, while retaining most of the model's performance.

Quant

quantization_config.bits in config.json reads 4 only because the Hub requires an integer. The bit width of each tensor comes from the weights. The Hub's "9B params" counts the packed tensors.

Performance (RTX 3090, int8 KV, MTP k=3)

Single stream. The prod-traffic column uses about 3.0 tokens per step, prod's measured MTP acceptance on mostly agentic coding traffic. The benchmark column uses the synthetic decode benchmark's prompt, which accepts about 3.9:

Eight concurrent requests (short prompts, long reasoning answers; 30 minutes of a knowledge benchmark on the served model):

The dips are prefills of new requests.

Quality

On a 50-task DeepSWE subset (Pi harness, one attempt per task) it solved 14/50, against 21/50 for stock Qwen3.8-27B quantized with the same 4.50 bpw recipe. MLA attention also costs more compute per forward pass than stock GQA. The gain is KV capacity: about 250k tokens of context on one 24 GB card.

Serving with vLLM

Both plugins are required: the MLA plugin provides the model architecture, and the EXL3 plugin branch loads the MLA latent projections as bf16. Tested on vLLM 0.27.1:

• Starwaves1/qwen-mla-vllm @ sm86-fp8: the MLA model, FP8 and int8 KV caches for sm_86
• Starwaves1/vllm-gsqrco-exl3 @ mla-latent-unquantized: EXL3 kernels (plugin-exl3/)

The settings used on an RTX 3090:

export QWEN_MLA_INT8_KV=1 QWEN_MLA_MAX_PAGE_RATIO=2 QWEN_MLA_DECODE_BACKEND=triton
vllm serve Starw1/Qwen3.8-27B-MLA-exl3-4.50bpw-H4 \
  --kv-cache-dtype fp8 --max-model-len 249400 --max-num-seqs 8 \
  --gpu-memory-utilization 0.94 --mamba-ssm-cache-dtype float16 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-prefix-caching --mamba-cache-mode align \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

QWEN_MLA_INT8_KV=1 stores the cache as int8 with a scale per token and decodes it on the int8 tensor cores. On a 3090 it decodes 1.5x faster than FP8 KV (ms/step at 110k context, 3 requests: 95.5 → 59.6), with lower KLD against bf16 KV. Without it, --kv-cache-dtype fp8 gives the FP8 cache. The int8 cache needs TP=1.

On GPUs with native FP8 tensor cores (sm_89 and newer: Ada, Hopper, Blackwell), the int8 cache's advantage goes away. There, TelperionAI's own builds with the official vLLM setup from the base model card are likely the better choice.