Muse-Glimmer-30B GPTQ 4-bit
GPTQ 4-bit (W4, group size 32) quantization of meta-models/Muse-Glimmer-30B
(revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68), a 29.8B dense text + vision model. The 52-layer text decoder is quantized
to INT4; the vision tower, vision adapter/projection, embeddings, norms and LM head stay in BF16. The checkpoint is 23 GB
(BF16 original: 59.6 GB).
Quantization Details
| Component | Precision | Notes |
|---|---|---|
self_attn.{q,k,v,gate,o}_proj |
INT4 GPTQ | all 52 decoder layers (incl. the attention output gate) |
mlp.{gate,up,down}_proj |
INT4 GPTQ | all 52 decoder layers |
| Vision tower, vision adapter, vision projection | BF16 | unchanged from the original |
| Embeddings, LM head, norms | BF16 |
416 quantized modules in total.
GPTQ configuration:
- Bits: 4
- Group size: 32
- Symmetric: Yes
- desc_act: No
- true_sequential: Yes
- mse: 2.0
- damp_percent: 0.05
- Fallback: RTN at a 0.5% calibration-coverage threshold
Calibration
- Dataset: mixed evol-codealpaca-v1 (code) + C4 (general English), the same set used for our other GPTQ releases
- Samples: 2048, binned uniformly across context lengths 256–2048 tokens, each passed as a single user turn through the model's chat template
- Quantizer: GPTQModel v7.3.5 (
muse_glimmermodel definition)
The exact quantization script is included as quantize.py.
Verification
Served with vLLM (tensor parallel 4, --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer):
- Text: correct, coherent answers (e.g. 17 × 23 →
391) - Image: read a newspaper front page correctly (masthead, printed date, lead headline)
- Tool calling: a weather tool request returned a parsed
get_weathercall with the right arguments - No server faults
Perplexity and fidelity against the BF16 original have not been measured yet.
Usage
vLLM
vllm serve btbtyler09/Muse-Glimmer-30B-GPTQ-4bit \
--tensor-parallel-size 4 \
--max-model-len 65536 \
--enable-auto-tool-choice --tool-call-parser muse_glimmer --reasoning-parser muse_glimmer
Reasoning strength
The chat template has no on/off thinking switch. It uses Reasoning strength: low | medium | high | xhigh (default high),
set in the system prompt or via chat_template_kwargs={"reasoning_strength": "low"}. The model's reasoning channel usually
opens by restating the user's request; this matches the BF16 original and is not caused by quantization.
License
Apache-2.0, inherited from the base model. The base model's USAGE_POLICY.md is included and applies to this checkpoint.