Muse-Glimmer-30B-Heretic · NVFP4 + DFlash (MXFP8 draft) · text-only

Download one folder and it runs. NVFP4 main model with three DFlash draft heads bundled in. Built to get 128k context and high throughput on two 16 GB consumer cards (TP2) at the same time.

Vision-capable variant → Muse-Glimmer-30B-Heretic-VL-NVFP4-DFlash-MXFP8

Measured (2×RTX 5070Ti 16GB · sm_120 · TP2 · vLLM 0.26.1)

★ The KV pool number is meaningful here. A 16 GB card is a constrained system: whatever is left after weights, draft head and activations is the KV pool. --kv-cache-memory 3200000000 is already close to the practical ceiling (~1.25 GiB/card is deliberately left free — without it the very first request OOMs). So it reflects what this hardware can do, not what I configured. (Contrast: the GB10 section below deliberately does not quote a KV pool — see why there.)

Long-prompt check: a 40,575-token input was answered correctly (6.2 s including prefill, measured on the same hardware on the previous revision).

⚠️ Scope of these numbers — please read before quoting them

Everything on this page is a short, single-domain measurement. It is not a long-running or broad benchmark:

• 7 prompts per configuration, single stream, one run each. No repeated runs for variance, no multi-hour stability testing.
• One prompt domain only: programming / technical Q&A. Chinese prose, long-form reports, agent / tool-calling traces, multi-turn chat and very-long-context continuation are all not covered.
• This matters especially for speculative decoding: DFlash acceptance is essentially a function of how redundant the text is. The same draft head can differ by more than 2× in accept_len across domains (in practice: tool-calling > code > Chinese reports > Chinese prose). So the ranking between the three draft heads below only holds for this one domain — it may well flip elsewhere.
• Concurrency figures are the theoretical value vLLM prints at startup (KV pool ÷ context), not throughput measured under real concurrent load.

If you are making a decision based on this, re-measure on your own prompt distribution.

Contents

model-0000{1,2}-of-00002.safetensors   16.98 GiB   main model
model.safetensors.index.json                       1,460 tensors / 2 files
config.json                                        has_vision: false
hf_quant_config.json                               modelopt MIXED_PRECISION
draft-mxfp8/                            2.76 GiB   DFlash draft head (default)
draft-nvfp4/                            1.80 GiB   DFlash draft head (smallest)
draft-bf16/                             4.76 GiB   DFlash draft head (upstream, unquantized)

Quantization recipe

How this recipe was decided

Not "quantize everything to 4-bit". Each component was decided separately, by one rule: open vLLM's model file and check whether that layer is constructed with a quant_config. If it isn't, it becomes a BF16 parameter — quantizing it guarantees a load failure.

Why NVFP4 and not FP8 for the backbone: vLLM's ModelOptFp8LinearMethod is a W8A8 path. It needs an input_scale, i.e. it needs calibration. There is no such thing as "weight-only FP8 without calibration" here — feed it uncalibrated FP8 and the model loads fine and then emits garbage. So if you want to skip calibration, your only options are NVFP4 (weight-only) or BF16.

Why group_size 16: it is W4A16_NVFP4's native block size, matching vLLM's kernel.

Three draft heads — pick one

This repo ships three DFlash draft heads. Switch by pointing --speculative-config's model at a different directory (or DRAFT=draft-nvfp4 bash run_example.sh):

These two columns are not the same thing:

• Hit rate = accepted draft tokens ÷ proposed draft tokens (K=15, so 15 proposed per round)
• accept_len = tokens actually emitted per round = 1 + hit_rate × K. This is the one that scales with speed.

The tok/s column was measured on a GB10 single card (same target, same K=15, same 7 prompts) and is only meant for ranking; for absolute numbers on 2×5070Ti see the table at the top. The last row is the same machine with speculative decoding switched off.

Two counter-intuitive results (bounded by the scope caveat above — one prompt domain only):

1. The BF16 draft head is not the best one. It is both larger and slower than MXFP8, and its acceptance is actually slightly lower (3.21 vs 3.32). Being 1.0 GiB/card larger also eats directly into the KV pool. "Unquantized must be more accurate" does not hold for draft heads.
2. NVFP4 costs acceptance, not peak speed (2.58 vs 3.32). It is still worth using — provided you spend the 0.48 GiB/card you saved on KV.

★ If you switch to draft-bf16/, lower --kv-cache-memory by about 2×10⁹ or it will OOM.

Running it

⚠️ The image must have muse_glimmer support — vllm/vllm-openai:latest will not start. Upstream vLLM has no muse_glimmer.py in main, v0.26.0, v0.26.1 or v0.27.0 (verified 2026-08-13; all four return 404).

Use the public build on Docker Hub, vllm/vllm-openai:muse-glimmer — vLLM 0.26.1rc1.dev608+g99a10304d · 2026-08-11 · multi-arch (amd64 + arm64):

vllm/vllm-openai@sha256:413c8fbecb1204a218117c77a4ea4b3a211d5686ff99c31d82ba6dd0cec8c5a6

Every command on this page and run_example.sh pin that digest, so a retagged image cannot silently change what you run. Switch back to latest once upstream merges support.

huggingface-cli download pottokao/Muse-Glimmer-30B-Heretic-NVFP4-DFlash-MXFP8 \
  --local-dir ./muse-text
bash run_example.sh ./muse-text        # or use the command below directly

The draft head lives in a subdirectory, so --speculative-config's model must be a local path — <download-dir>/draft-mxfp8 (or one of the other two). A repo id will not work.

docker run -d --name muse --gpus all --ipc host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e TORCH_CUDA_ARCH_LIST=12.0 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v $PWD/muse-text:/model -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai@sha256:413c8fbecb1204a218117c77a4ea4b3a211d5686ff99c31d82ba6dd0cec8c5a6 \
  serve /model --tensor-parallel-size 2 --generation-config auto \
  --quantization compressed-tensors --attention-backend TRITON_ATTN --kv-cache-dtype fp8 \
  --max-model-len 131072 --gpu-memory-utilization 0.95 --kv-cache-memory 3200000000 \
  --max-num-batched-tokens 2048 --max-num-seqs 4 --enable-chunked-prefill \
  --chat-template /model/chat_template.jinja \
  --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --enable-auto-tool-choice \
  --speculative-config '{"method":"dflash","model":"/model/draft-mxfp8","num_speculative_tokens":15,"attention_backend":"TRITON_ATTN"}' \
  --trust-remote-code

Dual-card configuration (2×16GB, as measured)

The parameters that matter above, and why each value:

Cost of K: DFlash is a block-diffusion drafter — one forward pass produces a whole 16-token block, so raising K costs almost no compute. It only costs CUDA-graph memory (K=4→15 is about +1.17 GiB). If memory is tight, K=8 still retains roughly 95% of the benefit.

Single large-memory card (NVIDIA GB10 / DGX Spark, as measured)

The same folder on a GB10 (sm_121, 121.6 GiB unified memory) single card — three changes:

docker run -d --name muse --gpus all --ipc host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v $PWD/muse-text:/model -p 8000:8000 --entrypoint vllm \
  vllm/vllm-openai@sha256:413c8fbecb1204a218117c77a4ea4b3a211d5686ff99c31d82ba6dd0cec8c5a6 \
  serve /model --tensor-parallel-size 1 --generation-config auto \
  --quantization compressed-tensors --attention-backend TRITON_ATTN --kv-cache-dtype fp8 \
  --max-model-len 131072 --gpu-memory-utilization 0.75 \
  --max-num-batched-tokens 4096 --max-num-seqs 16 --enable-chunked-prefill \
  --chat-template /model/chat_template.jinja \
  --reasoning-parser muse_glimmer --tool-call-parser muse_glimmer --enable-auto-tool-choice \
  --speculative-config '{"method":"dflash","model":"/model/draft-mxfp8","num_speculative_tokens":15,"attention_backend":"TRITON_ATTN"}' \
  --trust-remote-code

Measured on GB10:

No KV pool figure is quoted here on purpose. On GB10 the KV size is entirely determined by how much memory you hand it (util 0.75 above is just a test value), and such machines usually host other services too — publishing a number would invite reading it as a property of the model. Compute it yourself instead: subtract the three rows above (~26.7 GiB) and everything left can be KV; this model needs roughly 9,140 bytes/token with FP8 KV.

⇒ The two setups are opposites: dual 5070Ti buys speed (2.7×); GB10 buys "make the context as large as you like". TRITON_ATTN is equally mandatory on GB10 (sm_121 is neither SM90 nor SM100, so FP8 KV cannot use FlashAttention). If you don't need FP8 KV at all, GB10 has enough memory to just run --kv-cache-dtype auto and keep full precision.

★ Pitfalls already paid for (copying these will save you a day)

1. --attention-backend TRITON_ATTN is mandatory on sm_120, not a preference. FLASH_ATTN does not support FP8 KV (requires FA3 on SM90 or FA4 on SM100), and FP8 KV is the prerequisite for 128k on 16 GB cards — so TRITON is the only way in.
2. The drafter needs TRITON_ATTN too (set inside speculative-config; --attention-backend only covers the main model). flash_attn hits a CUDA device-side assert at K≥16.
3. Without --tool-call-parser muse_glimmer, chat completions come back empty.
4. Pass --kv-cache-memory explicitly rather than relying on util — and do not fill it: startup does not reserve for activations, so too large a value starts fine and OOMs on the first request.
5. K=15 is nearly free — DFlash produces a whole block per forward pass, so extra K only costs CUDA-graph memory (K=4→15 ≈ +1.17 GiB), not linear slowdown. Drop to K=8 under memory pressure and keep ~95% of the benefit.
6. If your vLLM build doesn't recognise DFlashDraftModel, change architectures in the draft's config.json to MuseGlimmerAssistantModel (the original is kept as config.json.orig).

Provenance and credits

• Base model: meta-models/Muse-Glimmer-30B (Meta Superintelligence Lab, Apache 2.0)
• DFlash draft head: the upstream Muse Glimmer DFlash assistant (draft-*/README_upstream.md is the original upstream model card)
• Abliteration source: mlasli/Muse-Glimmer-30B-Heretic-Abliterated-BF16, pinned at commit dabdf7eb48a770e4fda6a4bfc485ea21fae5fc84 — refusal-direction removal with Heretic over 500 Optuna trials, 6.5% refusals / KL 0.076. This repo only quantizes; it makes no capability changes.
• Quantization: NVIDIA TensorRT Model Optimizer 0.45.0

Use of this model is subject to the upstream Usage Policy.