bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

FP8 quantization of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B.

About this quantization

Native block-wise FP8 (E4M3): weights are quantized to FP8 with 128×128 block scales (weight_scale_inv) and activations are quantized dynamically per token — quant_method: "fp8", weight_block_size: [128, 128], the same DeepSeek-V3-style format that Qwen/Qwen3.6-27B-FP8 ships. vLLM and SGLang load it natively, and the model's MTP (multi-token-prediction) head is preserved, so self-speculative decoding works (≈3.2 accepted tokens/step). ≈29 GB instead of ≈55 GB bf16 — near-lossless quality at half the memory, with FP8 tensor-core throughput on Hopper / Blackwell (Ada / Ampere serve it via weight-only FP8-Marlin kernels).

Kept in bf16: lm_head, the MTP head, the vision tower, and the Gated-DeltaNet input gates (linear_attn.in_proj_a/b, whose 48-wide output doesn't tile a 128 block).

Usage

# vLLM
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

# SGLang
python -m sglang.launch_server --model-path bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 --trust-remote-code

Speculative decoding (MTP)

The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add --speculative-config when serving (requires vLLM ≥ 0.24.0):

vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.35 drafts per verify step here, for a further ≈2.4× on top of the finetune's token savings (see below).

For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF.

Expected performance

Measured on our internal serving harness on 8× RTX PRO 6000 (Blackwell) with vLLM 0.25.0, one configuration per GPU. Each cell is N=200 questions/dataset × 3 generation seeds over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); acc is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA).

FP8 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding ≈55% faster (36.4 vs 23.4 tok/s, MMLU-Pro standard), and MTP self-speculative decoding (≈3.1–3.3 accepted tokens per verify step) adds a further ≈1.8–2.0× — stacking with the finetune's token savings to ≈6× faster per task than the unquantized base (MMLU-Pro 13.7 s vs 97.3 s; RealWorldQA 6.8 s vs 34.2 s).

All timings are measured per request during the eval. median tokens = median completion length (the finetune's lever); tok/s = per-request steady-state decode rate (prefill excluded via time-to-first-token); task s = measured end-to-end wall-clock per request under the batch-size-16 concurrency; speedup = task s of the base model in standard decoding ÷ task s of the row. Qwen3.6-27B-FP8 (official) is Qwen/Qwen3.6-27B-FP8, the official FP8 quant of the base, shown for comparison.

MMLU-Pro (reasoning)

RealWorldQA (vision)