Muse-Glimmer-30B-GPTQ-INT4
Asymmetric INT4 weight-only (group_size=32) + GPTQ
A 4-bit quantization of meta-models/Muse-Glimmer-30B. ~2.6× smaller than BF16 with no measurable accuracy loss (≤0.2 pp average degradation across 8 text/reasoning benchmarks).
TL;DR
| value | |
|---|---|
| Format | INT4, asymmetric (zero-point), group_size=32, weight-only |
| Method | GPTQ (Hessian error compensation) + imatrix-mse observer, actorder=static |
| Calibration dataset | HuggingFaceH4/ultrachat_200k |
| Disk size | ~23 GB (vs ~60 GB BF16) |
| Avg accuracy degradation vs BF16 base | ≤0.2 pp across 8 text/reasoning benchmarks |
What was quantized (and what was protected)
| Component | Quantized | Protected (kept BF16) |
|---|---|---|
language_model (52 dense text layers) |
block linears — attention self_attn.{q_proj,k_proj,v_proj,o_proj,gate_proj} and FFN mlp.{gate_proj,up_proj,down_proj} (INT4, asymmetric, group_size=32) |
layer norms (input_layernorm, post_attention_layernorm, pre_feedforward_layernorm, post_feedforward_layernorm), model.language_model.embed_tokens, model.language_model.norm, lm_head |
vision_tower (frozen ViT perception encoder) |
— | entire model.vision_tower.* |
vision_adapter |
— | model.vision_adapter.fc1, model.vision_adapter.fc2 |
vision_projection |
— | model.vision_projection |
Evaluation
Same harness and config for base and quantized: lm-eval-harness, hf-multimodal
backend, greedy decoding, BF16. Each task is run on both the BF16 base and
this checkpoint, so every Δ is apples-to-apples.
| task | metric | base (BF16) | this (GPTQ-INT4) | Δ (pp) |
|---|---|---|---|---|
| gsm8k | exact_match | 92.72% | 93.03% | +0.31 |
| arc_easy | acc / acc_norm | 85.82 / 85.73 | 85.90 / 85.98 | +0.08 / +0.25 |
| arc_challenge | acc | 58.36% | 58.02% | −0.34 |
| hellaswag | acc | 53.50% | 53.10% | −0.40 |
| winogrande | acc | 78.06% | 78.30% | +0.24 |
| truthfulqa_mc1 | acc | 43.33% | 43.21% | −0.12 |
| minerva_math500 | exact_match | 57.80% | 57.40% | −0.40 |
| gpqa_diamond | acc | 41.41% | 40.91% | −0.51 |
| mmmu_val (6-subject subset) | acc (avg) | ~52.8% | ~50.0% | ~−2.8 |
Average degradation across the 8 text/reasoning tasks: ≤0.2 pp — lossless within task-eval noise. The MMMU 6-subject subset (~180 items) is reported separately; its per-subject deltas swing ±10 pp from sampling noise.
Serving
vllm serve abhishekchohan/Muse-Glimmer-30B-GPTQ-INT4 \
--enable-auto-tool-choice --tool-call-parser muse_glimmer \
--reasoning-parser muse_glimmer \
--generation-config auto
Requires vLLM nightly with Muse support (dedicated vllm/vllm-openai:muse-glimmer image).
License & attribution
Apache-2.0, inheriting the base model's license. Base model by meta-models.