Qwen3.6-27B-Fable-Fusion-711 — INT8 W8A16 + BF16 MTP

Offline W8A16 (INT8 weights, FP16/BF16 activations) of DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP for Ampere (2×RTX 3090) via vLLM compressed-tensors / Marlin.

This repo only changes numerics. All credit for the model to DavidAU (and collaborators). The original model card is preserved as BASE_MODEL_README.md.

Why W8A16 (not online FP8, not W4A16)

On sm_86 (RTX 3090) there is no FP8 tensor-core path. vLLM --quantization fp8 falls back to Marlin: FP8 weights unpacked to FP16 every GEMM. W8A16 uses the same Marlin weight-only path with per-channel/group INT8, which measured lower weight error than FP8-Marlin on this GPU (rel-L2 ≈ 0.006 vs ≈ 0.026 on random-Gaussian probes) at essentially identical decode throughput.

Format	Role on 2×3090 @ 262K	Notes
W8A16 (this repo)	Best default	Near-lossless weights; decode ≈ FP8-Marlin; fits 262K with selective GDN
Online `--quantization fp8`	Convenient	Same decode class, worse weight fidelity on Ampere
W8A8 INT8	Prefill play	Native INT8 GEMMs help cold TTFT; ~10% decode regression on skinny batches
W4A16 AutoRound	Memory/decode play	~19.5 GB; real long-context risk on hybrid GDN — see EMNLP’25 RULER@128K

We need 262144 context with MTP + vision on 48 GB. Keeping all linear_attn.* in BF16 (TheHouseOfTheDude / havenoammo “max safety” recipe) costs ~11 GB for GDN and only left ~2 GB KV (~97K). Selective GDN (quantize qkv/z/out, keep gates) lands at ~14.9 GiB/GPU weights → ~5.0 GiB KV → ~282K tokens.

Recipe

Tool: llm-compressor 0.12 · QuantizationModifier · scheme W8A16 · data-free RTN (no calibration set)
Format: compressed-tensors / pack-quantized (group INT8, memoryless_minmax)
Targets: Linear
Preserved BF16:
- lm_head
- model.visual.* (vision tower)
- mtp.* (grafted from the same DavidAU checkpoint — do not graft stock Qwen MTP onto a heretic finetune)
- linear_attn.in_proj_a / in_proj_b (GDN recurrent gates, out-dim 48)
- norms / conv1d / A_log / dt_bias (non-Linear; never packed)
Quantized INT8: MLP, full-attention projections, GDN in_proj_qkv / in_proj_z / out_proj
Size: ~31 GB (BF16 source ~55 GB)

# recipe.yaml
default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - lm_head
        - re:.*visual.*
        - re:.*mtp.*
        - re:.*linear_attn[.]in_proj_a$
        - re:.*linear_attn[.]in_proj_b$
      scheme: W8A16

Expected fidelity (KLD)

Sibling W8A16 of stock Qwen/Qwen3.6-27B with full linear_attn ignore (TheHouseOfTheDude/Qwen3.6-27B-INT8):

Mean KLD: 0.009441 nats/token (near-lossless)

On the same Fable-Fusion lineage, bf16 vs Q8_0 in a matched harness showed identical WikiText-2 PPL to four decimals (6.1978) — 8-bit is effectively free on this model when done carefully.

This checkpoint uses selective GDN quantization (required for 262K on 2×24 GB), so expect KLD in a similar ballpark, possibly slightly above the full-ignore Dude number. Measure on this repo with the script in the model card discussion / tools/measure_kld.py pattern: teacher = DavidAU BF16 logits, student = this W8A16, fixed prompts, mean KL(p_teacher || p_student) over next-token distributions (teacher-forced).

Measured on 2×RTX 3090 (this build)

Check	Result
`max_model_len`	262144
Weights / GPU (TP2)	14.85 GiB
Available KV (fp8_e4m3)	5.01 GiB (~282K tokens)
Smoke gen	Exact string match
MTP accept (coding sample)	72 / 75 draft tokens (96%) → ~3.9 tok/forward
Kernel path	`CompressedTensorsWNA16` → Marlin

Serve (2×24 GB)

vllm serve lued/Qwen3.6-27B-Fable-Fusion-711-INT8-W8A16-MTP \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.93 \
  --kv-cache-dtype fp8_e4m3 \
  --calculate-kv-scales \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --disable-custom-all-reduce \
  --trust-remote-code \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

On GeForce dual-GPU without NVLink, keep NCCL_P2P_DISABLE=1 — the driver reports P2P as unsupported (GNS); the flag is a no-op for throughput and avoids init stalls.

Base model (Nightmedia benches, instruct)

From the upstream card (mxfp8 / mxfp4 of this lineage):

	arc/c	arc/e	boolq	hswag	obkqa	piqa	wino
Fable-Fusion-711 mxfp8	0.711	0.879	0.910	0.790	0.514	0.823	0.763
Fable-Fusion-711 mxfp4	0.701	0.873	0.909	0.786	0.488	0.813	0.759
Qwen3.6-27B base mxfp8	0.647	0.803	0.910	0.773	0.450	0.806	0.742

W8A16 is intended to sit with the mxfp8 column, not the mxfp4 tradeoff.

Files

model-*-of-*.safetensors — packed W8A16 language / GDN GEMM weights (+ BF16 vision & gates)
model_mtp.safetensors — BF16 MTP head (~849 MB)
recipe.yaml — exact llm-compressor recipe
chat_template.jinja / chat_template-instruct.jinja — from upstream
BASE_MODEL_README.md — full DavidAU card

License

Apache-2.0 for this packaging. Respect upstream model terms for the base weights.

lued/Qwen3.6-27B-Fable-Fusion-711-INT8-W8A16-MTP