Qwythos-9B-v2-AWQ

AWQ (W4A16) quantization of empero-ai/Qwythos-9B-v2 — a 9.65B reasoning / uncensored model from Empero AI (qwen3_5: dense hybrid GatedDeltaNet linear-attention + full-attention over 32 text layers in a 3:1 pattern, plus a vision tower and an MTP head; multimodal-capable, 1M-token YaRN context). v2 preserves the deep chain-of-thought reasoning while eliminating the looping/degeneration of v1 (via FTPO).

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16. Quantized by: sahilchachra Tooling: llm-compressor (AWQModifier + QuantizationModifier) -> compressed-tensors pack-quantized

This is a quantized derivative. Weights, behavior, and license follow the base model — see the original card for full details, benchmarks, and citation.

What is quantized

Quantized to 4-bit:

• full-attention self_attn.{q,k,v,o}_proj
• mlp.{gate,up,down}_proj (all text layers)

Kept in BF16: GatedDeltaNet linear_attn (mamba) layers, vision tower (model.visual.*, 27 blocks), MTP head, token embeddings, lm_head, all norms (incl. q_norm / k_norm).

Calibration

AWQ: 128 sequences x 512 tokens of HuggingFaceH4/ultrachat_200k rendered through the model's own chat template. NVFP4 is data-free (not built here).

Prompt template & sampling

Uses the Qwen3.5 native chat template — ChatML (<|im_start|>role … <|im_end|>) with a <think>…</think> reasoning trace (a reasoning model). Apply it via tokenizer.apply_chat_template(messages, add_generation_prompt=True); do not hand-format prompts.

Recommended sampling: temperature=0.6, top_p=0.95, top_k=20, repetition_penalty=1.05, max_new_tokens=16384 (the model card's recommended settings; v2 no longer needs repetition_penalty as a band-aid, but the card still lists 1.05).

Usage (vLLM)

from vllm import LLM, SamplingParams

# This is a multimodal checkpoint: the vision tower is kept in BF16
# (only the text / MoE weights are 4-bit). vLLM builds the full model.
llm = LLM(
    model="sahilchachra/Qwythos-9B-v2-AWQ",
    trust_remote_code=True,
)
out = llm.chat(
    [{"role": "user", "content": "Hello!"}],
    SamplingParams(temperature=0.6, top_p=0.95, max_tokens=512),
)
print(out[0].outputs[0].text)

Serving via the CLI, pass the flag directly:

vllm serve sahilchachra/Qwythos-9B-v2-AWQ \
    --trust-remote-code \
    --max-model-len 262144 --reasoning-parser qwen3