Qwen3.8-27B Optimized MTP Heads (EXL3)

A per-tensor mixed-precision EXL3 quantization of the Qwen3.8-27B MTP (multi-token prediction) draft head, packaged as a sidecar that shadows the built-in MTP weights of the base model. Each optimized checkpoint is smaller than the same-size uniform quant while posting equal-or-better draft acceptance — because bits are concentrated where the draft head is actually sensitive (the feature projection fc) and spent far more cheaply where it is not (q, k, v, gate).

This is not a standalone model. It is the draft head a speculative-decoding server runs alongside the target: the MTP head proposes tokens, the target verifies them, and decoding stays lossless (accepted output is identical to the target's own output). What this changes is speed and memory — a smaller, higher-acceptance draft head means more tokens accepted per verification step and less VRAM for the drafter.

What the sidecar does

The files in this repo contain only the mtp.* tensors — the MTP head: the feature projection fc, one attention block (q/k/v/o), one SwiGLU MLP (gate/up/down), and the 7 norm tensors. When placed alongside a base model, exllamav3 loads the directory's safetensors in alphabetical order, and tensors in a later file shadow earlier ones. A file named mtp-*.safetensors sorts after the base model's model-*.safetensors, so this sidecar's mtp.* tensors override the base model's built-in MTP head.

Tip — trim the base model: because the sidecar shadows the base model's own MTP weights, those original mtp.* tensors are dead weight. You can ask your agent to strip the MTP weights from any base model you pair this sidecar with, reclaiming their full size with no change in behavior.

Compatible base models

The MTP head is defined by the shared Qwen3.8-27B base, so this sidecar works with any model built on that base, including:

• Qwen3.8-27B
• Swift
• ThinkingCap
• AREX-2
• abliterated variants
• any other Qwen3.8-27B derivative

Checkpoints

Measured performance

Draft acceptance (accepted draft tokens / total draft tokens), 50 runs per workload, against the Qwen3.8-27B target. Four workloads: repetitive, prose, prose_thinking (reasoning), code_thinking (reasoning). geomean is the geometric mean across the four — a size-balanced overall figure that weights every workload equally.

Mean accept length (draft tokens accepted per verification round) — the direct throughput driver (higher = fewer verification steps per token):

The mixed-precision win

Every optimized checkpoint sits on the size–quality Pareto frontier and dominates a uniform quant that is the same size or larger:

• mtp-3.7bpw (188.7 MiB) beats mtp-4bit (202.8 MiB) on all four workloads while being 14 MiB smaller (geomean 41.32 vs 40.28).
• mtp-4.4bpw (224.4 MiB) beats mtp-5bit (253.4 MiB) on repetitive, prose and prose_thinking while being 29 MiB smaller (geomean 41.81 vs 41.12).
• mtp-4.4bpw (224.4 MiB) exceeds mtp-6bit (304.0 MiB) on geomean (41.81 vs 41.54) while being 26% smaller.

The uniform 4/5-bit checkpoints are never the right answer: a smaller optimized checkpoint is always at least as good.

Acceptance vs size — Pareto frontier

Each plot shows draft acceptance against model size. The black line connects only the Pareto-optimal checkpoints; a checkpoint is dominated (gray open) if a smaller one is at least as good.

How it was built

Quantized clean from the BF16 MTP head (no dequant-from-quant) with exllamav3's EXL3 format: per-tensor bitrate, identity Hessian (no calibration data), seed 0, mul1. The 7 norm tensors are held in bf16 (a bit-exact copy of the source). The 8 quantizable linears are allocated independently:

The allocation comes from a systematic per-tensor sensitivity sweep: each tensor's bit width was lowered one at a time (the rest held at baseline) and the acceptance cost measured. The ordering that emerged is

fc  ≫  down  >  up  >  o  >  gate  >  k, v  >  q

fc is only 12% of the parameters but dominates acceptance (it has a hard cliff between 3 and 4 bit), so it is the first tensor to be protected and the last to be cut. q is 15% of the parameters but nearly free to compress, and k/v are quite insensitive but there's little to gain. Reinvesting the savings from q/k/v/gate into fc is exactly what makes each optimized checkpoint smaller than the uniform quant at equal acceptance.

Usage

Pair the sidecar with any compatible base model (see above) and run exllamav3 with MTP drafting enabled:

1. Copy one of the mtp-*.safetensors files into the base model's directory. Keep the mtp- prefix so it sorts after the base model's model-*.safetensors and shadows the built-in head.
2. (Optional) strip the base model's own mtp.* tensors to reclaim their drive space.
3. Launch with --mtp or configure mtp drafting in tabbyAPI.

License

Apache-2.0. Base model: Qwen/Qwen3.8-27B.