Muse-Glimmer-30B-Heretic-VL · NVFP4 + DFlash (MXFP8 draft) · with vision

Download one folder and it runs. NVFP4 main model, the vision tower's MLP quantized to NVFP4 as well, and three DFlash draft heads bundled in.

⚠️ This variant requires a small vLLM patch (shipped in vllm_patch/). See "Why the vision tower must be compressed" below. If you only need text → use Muse-Glimmer-30B-Heretic-NVFP4-DFlash-MXFP8 (no patch needed).

Measured (2×RTX 5070Ti 16GB · sm_120 · TP2 · vLLM 0.26.1)

★ The KV pool number is meaningful here. A 16 GB card is a constrained system: whatever remains after weights, vision tower, draft head and activations is the KV pool, and this is already near the practical ceiling. It reflects what this hardware can do, not what I configured. (Contrast: the GB10 section below deliberately does not quote one — see why there.)

Carrying vision costs almost no decode speed (92.8 vs 93.4) — the text decode path never reads the vision weights. The cost shows up purely in the KV pool (the vision tower takes 0.93 GiB/card). Both variants clear 3.8x concurrency at 131,072 context.

Vision correctness check: given a synthetic image, the model correctly reported a red rectangle outline on the left, a blue circle outline on the right, an off-white background and the position of two text lines; it OCR'd CANARY-7492 correctly, and spontaneously pointed out that the caption saying "red square" actually depicts a rectangle.

⚠️ Scope of these numbers — please read before quoting them

Everything on this page is a short, single-domain measurement. It is not a long-running or broad benchmark:

• 7 prompts per configuration, single stream, one run each. No repeated runs for variance, no multi-hour stability testing.
• One prompt domain only: programming / technical Q&A. Chinese prose, long-form reports, agent / tool-calling traces, multi-turn chat and very-long-context continuation are all not covered. Vision was verified for correctness, not benchmarked for throughput.
• This matters especially for speculative decoding: DFlash acceptance is essentially a function of how redundant the text is, and the same draft head can differ by more than 2× in accept_len across domains (in practice: tool-calling > code > Chinese reports > Chinese prose). The ranking between the three draft heads below only holds for this one domain.
• Concurrency figures are the theoretical value vLLM prints at startup (KV pool ÷ context), not throughput measured under real concurrent load.

If you are making a decision based on this, re-measure on your own prompt distribution.

★ Why the vision tower must be compressed (it's the entry ticket, not an optimization)

The vision tower is 3.58 GiB, so under TP2 you would expect 1.79 GiB per card. That is wrong.

MuseGlimmerVisionMLP is built from plain nn.Linear (muse_glimmer.py:650-651) — and plain nn.Linear has no tensor-parallel logic, so every card holds a full copy:

On 2×16 GB that leaves only 0.12 GiB/card for KV — not even one 131k sequence fits.

After the patch, the MLP becomes ColumnParallelLinear/RowParallelLinear, which buys quantizability and sharding at the same time: the vision tower drops from 3.14 to 0.93 GiB/card.

Text backbone quantization recipe

How this recipe was decided

Not "quantize everything to 4-bit". Each component was decided separately, by one rule: open vLLM's model file and check whether that layer is constructed with a quant_config. If it isn't, it becomes a BF16 parameter — quantizing it guarantees a load failure.

Why NVFP4 and not FP8 for the backbone: vLLM's ModelOptFp8LinearMethod is a W8A8 path. It needs an input_scale, i.e. calibration. There is no "weight-only FP8 without calibration" here — feed it uncalibrated FP8 and the model loads fine and then emits garbage. Skipping calibration leaves you NVFP4 (weight-only) or BF16.

Vision quantization recipe

Vision tower spec: 50 layers · hidden 1536 · intermediate 8960 · 16 heads. 100 modules were quantized (50 layers × mlp.fc1 + mlp.fc2) with W4A16_NVFP4 · group_size 16. Each module emits three tensors: weight (packed uint8), weight_scale (FP8 E4M3, one per 16 elements) and weight_scale_2 (F32, global). Biases stay BF16.

Result: vision shard 3.580 → 1.737 GiB; 3.14 → 0.93 GiB per card under TP2.

Quantization procedure (in case you want to reproduce it):

1. Container-style manual export, bypassing export_hf_checkpoint — it traces a forward pass, which weight-only quantization never needs. Instead: wrap each weight in an isolated nn.Linear, run mtq.quantize(..., forward_loop=None), then call _export_quantized_weight() per module.
2. ★Gate on a real scan before export: the number of modules actually quantized must equal 100, and so must the number of uint8 tensors — abort otherwise. When modelopt's quant_cfg is given as a list, later rules silently override earlier ones, so exclusions placed first get swallowed. Only counting the real result catches this.
3. Emit the new model-vision-nvfp4.safetensors and update quantized_layers in config.json (★using vLLM-side names — see "Two easy traps" below).

Running it

⚠️ The image must have muse_glimmer support — vllm/vllm-openai:latest will not start. Upstream vLLM has no muse_glimmer.py in main, v0.26.0, v0.26.1 or v0.27.0 (verified 2026-08-13; all four return 404).

Use the public build on Docker Hub, vllm/vllm-openai:muse-glimmer — vLLM 0.26.1rc1.dev608+g99a10304d · 2026-08-11 · multi-arch (amd64 + arm64):

vllm/vllm-openai@sha256:413c8fbecb1204a218117c77a4ea4b3a211d5686ff99c31d82ba6dd0cec8c5a6

Every command on this page and run_example.sh pin that digest.

huggingface-cli download pottokao/Muse-Glimmer-30B-Heretic-VL-NVFP4-DFlash-MXFP8 \
  --local-dir ./muse-vl

1. Prepare the patch (required)

vllm_patch/muse_glimmer.py is the result of 8 edits applied to vLLM 0.26.1's muse_glimmer.py. If your vLLM version differs, do not use it as-is — regenerate it from your own image with the patcher in the same directory:

docker run --rm -v $PWD:/host --entrypoint python3 <your vllm image> \
  /host/muse-vl/vllm_patch/patch_muse_vision_quant.py /host/muse_glimmer_patched.py

The patcher asserts that each of its edits matches exactly once, and calls sys.exit otherwise — so a version change fails loudly instead of silently patching the wrong place.

2. Start it

docker run -d --name muse-vl --gpus all --ipc host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e TORCH_CUDA_ARCH_LIST=12.0 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v $PWD/muse-vl:/model \
  -v $PWD/muse-vl/vllm_patch/muse_glimmer.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/muse_glimmer.py:ro \
  -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai@sha256:413c8fbecb1204a218117c77a4ea4b3a211d5686ff99c31d82ba6dd0cec8c5a6 \
  serve /model --tensor-parallel-size 2 --generation-config auto \
  --quantization compressed-tensors --attention-backend TRITON_ATTN --kv-cache-dtype fp8 \
  --max-model-len 131072 --gpu-memory-utilization 0.95 --kv-cache-memory 2300000000 \
  --max-num-batched-tokens 2048 --max-num-seqs 4 --enable-chunked-prefill \
  --chat-template /model/chat_template.jinja \
  --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --enable-auto-tool-choice \
  --speculative-config '{"method":"dflash","model":"/model/draft-mxfp8","num_speculative_tokens":15,"attention_backend":"TRITON_ATTN"}' \
  --trust-remote-code

Three draft heads — pick one

DRAFT=draft-nvfp4 bash run_example.sh switches between them:

The two columns mean different things: hit rate = accepted ÷ proposed draft tokens (K=15); accept_len = tokens emitted per round = 1 + hit_rate × K, and that is the one that scales with speed.

tok/s was measured on a GB10 single card (same target, same K=15, same 7 prompts) and is only for ranking; absolute numbers for 2×5070Ti are at the top, and the last row is the same machine with speculative decoding off.

★ The BF16 draft head is not the best one — bigger and slower than MXFP8, with slightly lower acceptance (3.21 vs 3.32). ★ This variant is already tighter on memory; switching to draft-bf16/ needs --kv-cache-memory lowered by roughly 2×10⁹. See the text-only README for the full trade-off.

Contents

model-0000{1,2}-of-00002.safetensors   16.98 GiB   main model (text)
model-vision-nvfp4.safetensors          1.74 GiB   vision tower (MLP quantized to NVFP4)
model.safetensors.index.json                       2,469 tensors / 3 files
config.json                                        517 quantization entries (100 of them vision)
draft-mxfp8/                            2.76 GiB   DFlash draft head (default)
draft-nvfp4/                            1.80 GiB   DFlash draft head (smallest)
draft-bf16/                             4.76 GiB   DFlash draft head (upstream, unquantized)
vllm_patch/                                        patched file + the patcher that generates it

Dual-card configuration (2×16GB, as measured)

Single large-memory card (NVIDIA GB10 / DGX Spark, as measured)

The same folder on a GB10 (sm_121, 121.6 GiB unified memory) single card. ★A single card has no "MLP fully replicated" problem, but the patch is still required — the weights are already packed NVFP4, and an unpatched nn.Linear cannot accept them (shape error at load). Three changes:

docker run -d --name muse --gpus all --ipc host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v $PWD/muse-vl/vllm_patch/muse_glimmer.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/muse_glimmer.py:ro \
  -v $PWD/muse-vl:/model -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai@sha256:413c8fbecb1204a218117c77a4ea4b3a211d5686ff99c31d82ba6dd0cec8c5a6 \
  serve /model --tensor-parallel-size 1 --generation-config auto \
  --quantization compressed-tensors --attention-backend TRITON_ATTN --kv-cache-dtype fp8 \
  --max-model-len 131072 --gpu-memory-utilization 0.75 \
  --max-num-batched-tokens 4096 --max-num-seqs 16 --enable-chunked-prefill \
  --chat-template /model/chat_template.jinja \
  --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --enable-auto-tool-choice \
  --speculative-config '{"method":"dflash","model":"/model/draft-mxfp8","num_speculative_tokens":15,"attention_backend":"TRITON_ATTN"}' \
  --trust-remote-code

Measured on GB10 (figures below are for the text-only variant; the vision variant was not separately benchmarked on GB10, but the vision tower only adds 1.74 GiB of weights, which is negligible against 121 GiB):

No KV pool figure is quoted here on purpose. On GB10 the KV size is entirely determined by how much memory you hand it (util 0.75 above is just a test value), and such machines usually host other services too — publishing a number would invite reading it as a property of the model. Compute it yourself: subtract the three rows above (~26.7 GiB) and everything left can be KV; this model needs roughly 9,140 bytes/token with FP8 KV.

⇒ The two setups are opposites: dual 5070Ti buys speed (2.7×); GB10 buys "make the context as large as you like". TRITON_ATTN is equally mandatory on GB10 (sm_121 is neither SM90 nor SM100).

★ Two easy traps

1. Vision cannot be switched off with has_vision: false alone. AutoWeightsLoader(self) in muse_glimmer.py is constructed without skip_prefixes, so the model.vision_tower.* tensors in the checkpoint find no home and it fails outright:

ValueError: There is no module or parameter named 'vision_encoder' in MuseGlimmerForCausalLM

Configuration can disable the compute path, not the weight loading. To actually save the memory, use the text-only repo — it keeps the vision tower in a dedicated shard that its index does not list, and since vLLM filters at file granularity that file is never even opened (the HF equivalent of GGUF's external mmproj).

2. quantized_layers keys use vLLM-side names, not checkpoint names. ModelOptMixedPrecisionConfig.apply_vllm_mapper() runs the keys through the weight mapper, but the mapper's substring rules require a dot on both sides (".mlp.fc1." → ".mlp.c_fc."), and a module name has no trailing dot. Write checkpoint names and they will not be translated — the layer silently falls back to Unquantized and then explodes on shape mismatch at load. So the vision entries here are written as vision_encoder.transformer.N.mlp.c_fc, while the tensor names keep their original checkpoint form (they end in .weight, which the rules do match).

Other pitfalls (why TRITON_ATTN is mandatory, the drafter backend, kv-cache-memory, cost of K) are identical to the text-only variant — see that README.

Provenance and credits

• Base model: meta-models/Muse-Glimmer-30B (Meta Superintelligence Lab, Apache 2.0)
• DFlash draft head: the upstream Muse Glimmer DFlash assistant (draft-*/README_upstream.md is the original upstream model card)
• Abliteration source: mlasli/Muse-Glimmer-30B-Heretic-Abliterated-BF16, pinned at commit dabdf7eb48a770e4fda6a4bfc485ea21fae5fc84 — refusal-direction removal with Heretic over 500 Optuna trials, 6.5% refusals / KL 0.076. This repo only quantizes; it makes no capability changes.
• Quantization: NVIDIA TensorRT Model Optimizer 0.45.0

Use of this model is subject to the upstream Usage Policy.