Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-W4A16-AWQ

W4A16 (4-bit weights, 16-bit activations) AWQ compressed-tensors quantization of DavidAU/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP.

The source model has since shipped updated jinja chat templates; chat_template.jinja here is the current version from the source repo.

Quantization method

• Scheme: W4A16_ASYM — 4-bit asymmetric per-group quantization (group size 128) of all Linear weights, stored in the compressed-tensors pack-quantized format (weight_packed / weight_scale / weight_zero_point / weight_shape), which LMDeploy turbomind auto-detects and loads natively (including the MTP heads and vision tower, which stay BF16).
• Tooling: llmcompressor one-shot offline quantization with CPU offloading (compressed_tensors.offload.load_offloaded_model), so the full-precision source fits on a 2×16 GB VRAM setup.
• AWQ activation smoothing: AWQModifier with the layer-scoped hybrid-attention mappings from build_hybrid_attention_mappings — full-attention input_layernorm → self_attn.q/k/v, post_attention_layernorm → mlp.gate/up, and mlp.up_proj → mlp.down_proj, with duo_scaling="both" and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.
• Unquantized (kept BF16): embeddings, lm_head, norms, linear_attn.in_proj_a/b, the vision tower, and MTP heads.

Benchmarks

See the original model's README for full benchmark tables.

Usage

Tested with LMDeploy turbomind:

from lmdeploy import pipeline, TurbomindEngineConfig

pipe = pipeline(
    "TheUnderscore/Qwen3.5-9B-The-Defiant-Fable-Uncensored-Heretic-NEO-IMATRIX-MAX-MTP-W4A16-AWQ",
    backend_config=TurbomindEngineConfig(
        tp=1,
        model_format="compressed-tensors",
        language_model_only=True,
    ),
)
print(pipe("Hello, who are you?").text)

Files

• chat_template.jinja — the current chat template from the source repo. Also included: chat_template-toolcall.jinja (enhanced tool-calling) variant. This isn't referenced in tokenizer_config.json, so the loader uses its own default. To switch, swap the contents of chat_template.jinja with a variant (or point your server's chat-template option at the file).
• quantize-awq-hybrid.py — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered -of-N shards). --offload_dir selects where per-rank CPU offload temp folders live (defaults to the current working directory).
• model-nonquant.safetensors — unquantized tensors (mtp.* and model.visual.*) preserved BF16 so the full model architecture is loadable.