Muse-Glimmer-30B-NVFP4-AutoRound

NVFP4 · W4A4 — 4-bit weights + 4-bit activations. Quantization of meta-models/Muse-Glimmer-30B produced with Intel AutoRound, packaged in compressed-tensors for vLLM on NVIDIA Blackwell. The W4A4 scheme trades a little accuracy for FP4 activation speed on Blackwell. Text decoder is NVFP4 W4A4; vision tower + lm_head stay BF16. ~22 GB (vs ~60 GB BF16).

→ Want any-GPU support and higher accuracy? Use the W4A16 sibling (int4, weights-only).

Model overview

• Base model: meta-models/Muse-Glimmer-30B — a dense (Gemma2-derived) multimodal decoder + vision tower.
• Quantization: NVFP4 (W4A4) on the text-decoder Linear layers; vision tower / adapter / projection / patch-embedder / lm_head kept BF16.
• Format: compressed-tensors (nvfp4-pack-quantized), auto-detected by vLLM.
• Quantizer: Intel AutoRound (arXiv:2309.05516).
• Intended use: efficient inference on NVIDIA Blackwell (B200, RTX PRO 6000, RTX 5090, DGX Spark). Blackwell has native FP4 tensor cores, so a W4A4 model runs its matmuls directly in 4-bit — less memory traffic and faster compute than 16-bit or FP8.

Quantization recipe

• Method: AutoRound, scheme NVFP4 — 4-bit weights and input activations, NVFP4 microscale (group size 16, FP8 e4m3 block scale + FP32 global), symmetric.
• Quantized: all Linear in the 52 model.language_model.layers.* decoder blocks.
• Kept BF16: model.vision_tower.*, model.vision_adapter.*, model.vision_projection, patch_embedder, lm_head (mirrors the RedHatAI/Muse-Glimmer-30B-NVFP4 scope).
• Calibration: NeelNanda/pile-10k, 128 samples, seqlen 2048, 200 tuning iters.
• Cost: ~94 minutes, peak ~47.6 GB VRAM.

Deployment (vLLM)

Recommended image: vllm/vllm-openai:muse-glimmer. vLLM auto-detects the quant scheme from config.json — no quantization flag needed.

vllm serve dbirks/Muse-Glimmer-30B-NVFP4-AutoRound \
  --served-model-name muse-glimmer \
  --max-model-len 8192 \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Example compose.yaml

services:
  muse-glimmer:
    image: vllm/vllm-openai:muse-glimmer
    ports:
      - "8000:8000"
    ipc: host
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command:
      - "--model=dbirks/Muse-Glimmer-30B-NVFP4-AutoRound"
      - "--served-model-name=muse-glimmer"
      - "--max-model-len=8192"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser=muse_glimmer"
      - "--reasoning-parser=muse_glimmer"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Note: Muse-Glimmer is a reasoning model — with the reasoning parser on, chain-of-thought comes back in the reasoning field and the final answer in content; give it enough max_tokens. (Older vLLM builds on consumer Blackwell/SM120 may need VLLM_ATTENTION_BACKEND=FLASHINFER; the recommended image above does not.)

Evaluation — accuracy recovery vs BF16

Averaged over the OpenLLM-v1 suite (EleutherAI lm-evaluation-harness, in-process vllm backend), recovery = quant ÷ BF16 × 100.

(Same setup, for reference: Red Hat GPTQ-NVFP4 = 98.4%; our int4 W4A16 sibling = 98.5%.)

Hardware & format notes

• NVFP4 requires NVIDIA Blackwell (SM100 / SM120 / SM121) FP4 tensor cores — it won't accelerate on Ada/Hopper.
• Only the text decoder is quantized; the vision tower stays BF16 (intentional, to preserve multimodal quality).

Reproducibility

from auto_round import AutoRound
ar = AutoRound(
    "meta-models/Muse-Glimmer-30B",
    scheme="NVFP4", dataset="NeelNanda/pile-10k",
    nsamples=128, seqlen=2048, batch_size=4, iters=200,
    device_map=0, trust_remote_code=True, quant_nontext_module=False, seed=42,
)
ar.quantize_and_save(output_dir="Muse-Glimmer-30B-NVFP4-AutoRound", format="llm_compressor")

Toolchain: auto-round 0.15.0, transformers 5.16.0.dev0 (from source — required for the muse_glimmer arch), compressed-tensors 0.17.0, torch 2.11.0+cu130.

Citation

@article{cheng2023optimize,
  title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}