Muse-Glimmer-30B-int4-AutoRound

int4 (W4A16) quantization of meta-models/Muse-Glimmer-30B, produced with Intel AutoRound and packaged in compressed-tensors for vLLM. Runs on any modern NVIDIA GPU (Ampere → Ada → Hopper → Blackwell) via Marlin — 4-bit weights, BF16 activations. Text-only quant: the transformer decoder is int4; vision tower + lm_head stay BF16. ~20.7 GB on disk (vs ~60 GB BF16).

Model overview

• Base model: meta-models/Muse-Glimmer-30B — a dense (Gemma2-derived) multimodal reasoning decoder + vision tower.
• Quantization: W4A16 (int4 weight-only, group size 128) on the text-decoder Linear layers; vision tower / adapter / projection / patch-embedder / lm_head kept BF16.
• Format: compressed-tensors (pack-quantized), auto-detected by vLLM.
• Quantizer: Intel AutoRound (arXiv:2309.05516), 128 samples · seqlen 2048 · 200 iters · NeelNanda/pile-10k.
• Why this one: highest accuracy recovery of the 4-bit variants and the broadest hardware support — if you don't have a Blackwell card, this is the one to use. (For Blackwell FP4 speed, see the NVFP4 sibling.)

Evaluation — accuracy recovery vs BF16

EleutherAI lm-evaluation-harness, in-process --model vllm, OpenLLM-v1, identical settings; Recovery % = quant ÷ BF16 × 100. (No chat template: Muse is a reasoning model, and chat-template MCQ loglikelihood collapses to ~random — see notes.)

_{For reference on the same harness: Red Hat GPTQ-NVFP4 = 98.36%, AutoRound NVFP4 (W4A4) = 97.09%. This W4A16 build is the highest-recovery 4-bit option, at the smallest size (20.7 GB).}

Deployment (vLLM)

vLLM auto-detects the scheme from config.json — no quantization flag needed. int4 W4A16 serves via the Marlin kernel on any Ampere-or-newer GPU.

vllm serve dbirks/Muse-Glimmer-30B-int4-AutoRound \
  --tensor-parallel-size 1 --max-model-len 8192 \
  --enable-auto-tool-choice --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
r = client.chat.completions.create(model="dbirks/Muse-Glimmer-30B-int4-AutoRound",
    messages=[{"role":"user","content":[
        {"type":"text","text":"Describe this image."},
        {"type":"image_url","image_url":{"url":"https://…"}}]}])
print(r.choices[0].message.content)

Note: Muse-Glimmer is a reasoning model — with the reasoning parser on, chain-of-thought comes back in the reasoning field and the final answer in content; give it enough max_tokens.

Limitations

• Small accuracy loss vs BF16 (see table). Only the text decoder is quantized (vision tower BF16, intentional).
• Weight-only (W4A16): activations stay BF16, so no activation-quant speedup — but maximal compatibility + accuracy. For Blackwell FP4 (W4A4) speed, use the NVFP4 sibling.

Reproducibility

from auto_round import AutoRound
ar = AutoRound("meta-models/Muse-Glimmer-30B", scheme="W4A16", dataset="NeelNanda/pile-10k",
               nsamples=128, seqlen=2048, batch_size=4, iters=200,
               device_map=0, trust_remote_code=True, quant_nontext_module=False, seed=42)
ar.quantize_and_save(output_dir="Muse-Glimmer-30B-int4-AutoRound", format="llm_compressor")

Toolchain: auto-round 0.15.0, transformers 5.16.0.dev0 (source — required for the muse_glimmer arch), compressed-tensors 0.17.0, torch 2.11.0+cu130. Full effort log: home-k8s issue #81.

Citation

@article{cheng2023optimize, title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs}, author={Cheng, Wenhua and others}, journal={arXiv:2309.05516}, year={2023}}