Qwopus3.6-27B-v2 · HLWQ Q5

Hadamard-Lloyd Weight Quantization (Q5) of Jackrong/Qwopus3.6-27B-v2 — a Qwen3.5-based reasoning-distilled model with hybrid attention, MTP head, and vision tower.

---

Benchmarks

Measured on a single RTX 6000 Pro Blackwell (96 GB) with transformers main, BF16 path, SDPA attention.

Inference speed

Weight compression

KV cache compression

MTP head accuracy (preservation under quantization)

The MTP head is loaded raw BF16 from the source checkpoint and queried against the HLWQ-quantized main model. 79% top-1 acceptance means the MTP draft matches the quantized model's greedy output for ~4 of every 5 tokens — strong evidence that Q5 + INT4 preserved the semantic surface that MTP was trained against.

---

Architecture

• Qwen3_5ForConditionalGeneration — hybrid linear+full attention, multimodal, MTP head
• 64 text layers: 48 linear_attention + 16 full_attention (pattern [L,L,L,F]×16)
• head_dim=256, hidden_size=5120, GQA (24 Q heads / 4 KV heads)
• 1 MTP layer (mtp_num_hidden_layers=1) for speculative decoding
• Vision tower kept in BF16 (multi-modal projector untouched)
• 496 Linears PQ5-quantized → 400 of those converted to torchao INT4 (non-128-aligned shapes stay PQ5-dequant BF16)

Pipeline

BF16 weights (58 GB)
        │
        ▼
HLWQ Q5: Hadamard rotation + Lloyd-Max 32-centroid quantization (block=128)
        │
        ▼
Dequantize per-Linear → BF16
        │
        ▼
torchao Int4WeightOnlyConfig(group_size=128) on aligned text Linears
        │
        ▼
HLWQ Q3 KV cache during generation (bit-packed, head_dim=256)
        │
        ▼
Generation at ~23 tok/s on consumer-class GPU

Quick Start

pip install polarquant

import polarengine_vllm  # auto-registers with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "caiovicentino1/Qwopus3.6-27B-v2-HLWQ-Q5"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, device_map="auto", dtype="bfloat16")

inputs = tok.apply_chat_template(
    [{"role": "user", "content": "Explain quantum entanglement in two paragraphs."}],
    add_generation_prompt=True, return_tensors="pt", return_dict=True,
).to(model.device)

out = model.generate(**inputs, max_new_tokens=300, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

GPU sizing

Methodology notes

• Weight quantization: each nn.Linear is rotated by a 128×128 Walsh-Hadamard matrix, scaled by √128, then assigned to the nearest of 32 Lloyd-Max centroids (computed for a unit Gaussian). Codes are bit-packed (5 bits → 0.625 byte/weight).
• KV cache quantization: per-head Hadamard 256×256 + 8-centroid (Q3) Lloyd-Max with bit-packing (3 bits / scalar). A 128-token residual buffer holds recent tokens unquantized for accuracy.
• Hybrid handling: the 48 linear-attention layers receive a hybrid cache layer passthrough (no KV quantization needed — recurrent state instead).
• Multimodal prefix: weights re-prefixed with model.language_model. to match the base model's safetensors key layout.

References

• Paper: arXiv:2603.29078
• Code: github.com/caiovicentino/eoq-quantization
• Base model: Jackrong/Qwopus3.6-27B-v2
• Original GGUF release: Jackrong/Qwopus3.6-27B-v2-MTP-GGUF