[!IMPORTANT] Naming notice (2026-04-10). The "HLWQ" technique used in this model is being rebranded to HLWQ (Hadamard-Lloyd Weight Quantization). The change is only the name; the algorithm and the weights in this repository are unchanged.

The rebrand resolves a name collision with an unrelated, earlier KV cache quantization method also named HLWQ (Han et al., arXiv:2502.02617, 2025). HLWQ addresses weight quantization with a deterministic Walsh-Hadamard rotation and Lloyd-Max scalar codebook; Han et al.'s HLWQ addresses KV cache quantization with a random polar rotation. The two methods are technically distinct.

Existing loaders that load this repository by ID continue to work without changes. Future model uploads will use the HLWQ name.

Reference paper for this technique: arXiv:2603.29078 (v2 in preparation; v1 still uses the old name).

Qwopus-MoE-35B-A3B — HLWQ INT4 CompressedTensors

35B Mixture-of-Experts quantized to INT4 — loads natively in vLLM with Marlin kernel. Zero plugins, zero custom code.

---

Quick Start

Standard (vLLM native)

pip install vllm

vllm serve caiovicentino1/Qwopus-MoE-35B-A3B-HLWQ-Q5 \
  --language-model-only \
  --enforce-eager \
  --max-model-len 4096

With Expert Offloading (fits RTX 4090!)

pip install git+https://github.com/caiovicentino/vllm-expert-offload.git

vllm serve caiovicentino1/Qwopus-MoE-35B-A3B-HLWQ-Q5 \
  --language-model-only \
  --enforce-eager \
  --moe-expert-cache-size 8 \
  --max-model-len 4096

37.4 tok/s with LFRU expert cache — only 8 hot experts in GPU, rest on CPU. 1.58x faster than all-in-GPU and fits in ~8 GB VRAM.

Python API

from vllm import LLM, SamplingParams

llm = LLM(
    model="caiovicentino1/Qwopus-MoE-35B-A3B-HLWQ-Q5",
    trust_remote_code=True,
    enforce_eager=True,
    max_model_len=4096,
)

output = llm.generate(
    ["Explain quantum computing in simple terms."],
    SamplingParams(max_tokens=256, temperature=0.7),
)
print(output[0].outputs[0].text)

---

Validated Deploy (April 2026, updated vLLM)

The original --language-model-only flag has been removed from recent vLLM builds. Current vLLM (0.17.1rc1+) requires a slightly different invocation because the model config declares Qwen3_5MoeForConditionalGeneration (multimodal class) even though this repo ships text-only weights.

Working command (current vLLM + fork feature branch)

pip install git+https://github.com/caiovicentino/vllm-expert-offload.git@feat-ct-moe-wna16-marlin-offload

vllm serve caiovicentino1/Qwopus-MoE-35B-A3B-HLWQ-Q5 \
  --served-model-name qwopus-ct \
  --moe-expert-cache-size 128 \
  --enforce-eager \
  --trust-remote-code \
  --max-model-len 4096 \
  --limit-mm-per-prompt '{"image":0,"video":0}' \
  --gpu-memory-utilization 0.90

Key flag changes vs older instructions:

• --language-model-only → removed (flag no longer exists in current vLLM).
• --limit-mm-per-prompt '{"image":0,"video":0}' → new, tells vLLM not to allocate dummy image tokens in the profile-run. Without this the engine tries to load a fake image through a vision encoder that has no weights, and crashes during KV-cache sizing.
• git+.../vllm-expert-offload.git@feat-ct-moe-wna16-marlin-offload → feature branch with:
  • opt-in expert LRU/LFRU cache for CompressedTensorsWNA16MarlinMoEMethod (first CT INT4 MoE method in the fork with expert offload support)
  • disk-backed backing store via torch.from_file(shared=True) — zero pinned CPU RAM, OS page cache handles hot pages
  • correct overflow fallback for top_k=8 models (old truncation produced silent garbage when unique experts > cache size)

Cache size tuning

This model is top_k=8 with 256 experts, so a single prefill forward can activate 40–60 unique experts. The cache policy is per-forward:

• --moe-expert-cache-size 128 — recommended. All hot experts fit, ~24 tok/s on RTX PRO 6000, ~8 GB persistent LRU buffer on GPU.
• --moe-expert-cache-size 64 — tighter GPU footprint, ~20 tok/s, occasional overflow fallbacks.
• --moe-expert-cache-size 8 — minimum footprint, ~10 tok/s, every forward hits the overflow fallback (correct but copies CPU→GPU every call). Useful on 12 GB VRAM + accepting the speed cost.

For top-k=2 models (Nemotron, Gemma-4-26B-A4B) --moe-expert-cache-size 8 is fast because those only activate ~4-8 unique experts per forward.

Repo files added for vLLM loading

• preprocessor_config.json — stub pulled from Qwen/Qwen2-VL-7B-Instruct. vLLM Qwen3-VL processor init requires this file even when images are disabled. Never actually invoked at text inference.
• tokenizer_config.json — tokenizer_class set to Qwen2TokenizerFast (was PreTrainedTokenizerFast), required by the Qwen3-VL processor class check.

Validated on

NVIDIA RTX PRO 6000 Blackwell Server Edition (102 GB VRAM, 176 GB host RAM, 236 GB SSD). Disk usage during deploy: ~20 GB HF cache + ~31 GB disk-backed expert cache = ~51 GB peak. Host RAM stable at ~7 GB used, ~167 GB in OS page cache (the disk-backed mmap pages, reclaimable). Outputs coherent on 3-prompt smoke test at cache sizes 8, 64, 128. Informal PPL sanity check: 9.6 on a 357-token wikitext-style sample with cache=8 (the authoritative PPL 6.56 above is from the full WikiText-2 evaluation with a different config).

---

Architecture

---

Quantization Details

Format: CompressedTensors pack-quantized — industry standard, native vLLM support via Marlin kernel.

Pipeline: Base BF16 → INT4 symmetric group128 → pack INT32 → CompressedTensors safetensors

---

Benchmarks

Perplexity (WikiText-2, Full Test Set)

Evaluated over the complete WikiText-2 test set (1,155 windows, 295,680 tokens):

PPL converges to 6.56 on full WikiText-2. We have not measured the BF16 baseline PPL for this specific MoE — we can only say this is a low-PPL result on the benchmark, without a direct BF16 comparison.

Generation Speed

Tested on NVIDIA RTX PRO 6000 Blackwell (102 GB VRAM):

The LFRU expert cache is faster because only 8 hot experts are in GPU — better memory locality, less fragmentation across 256 experts.

2.3x faster than BF16 with 3.7x smaller download and 9x less VRAM.

---

HLWQ Ecosystem

All models load natively in vLLM — no plugins required:

---

GPU Requirements

Expert LFRU offloading via vllm-expert-offload enables running this 35B MoE on consumer GPUs (RTX 4090, 3090). Only hot experts stay in GPU, rest on CPU. Counterintuitively, this is faster than all-in-GPU due to better memory locality.

Required Flags

---

Technical Notes

• Expert format: Per-expert 2D tensors (experts.{N}.gate_proj.weight_packed), not 3D stacked. vLLM FusedMoE handles stacking internally.
• Reference format: Follows RedHatAI/Qwen3-30B-A3B-quantized.w4a16 conventions.
• Marlin kernel: Fused INT4 dequant + matmul for maximum throughput.
• Vision weights: Excluded (use --language-model-only). The config retains vision_config for architecture resolution.

---

Links

• Paper: HLWQ — Hadamard-Rotated Post-Training Quantization
• GitHub: polarengine-vllm
• Expert Offloading Fork: vllm-expert-offload — LFRU expert cache for consumer GPUs
• Base Model: samuelcardillo/Qwopus-MoE-35B-A3B
• Reference Format: RedHatAI/Qwen3-30B-A3B-quantized.w4a16

Citation

@article{vicentino2026polarquant,
  title={HLWQ: Hadamard-Rotated Post-Training Quantization},
  author={Vicentino, Caio},
  journal={arXiv preprint arXiv:2603.29078},
  year={2026}
}