Muse-Glimmer-30B-NVFP4-GGUF

A family of seven GGUF files of Muse-Glimmer-30B (the dense 30B vision-language model from Meta, 52 transformer layers, sliding window attention with a [Local,Local,Local,Global] period-4 pattern, 131,072-token native context), converted from the Red Hat AI NVFP4 checkpoint RedHatAI/Muse-Glimmer-30B-NVFP4.

All seven tiers share a byte-identical backbone of 1,561 tensors (416 NVFP4 attention/MLP weights + 1,145 F32 norms/scales/QK-norm vectors, 13.35 GB) and differ only in the lm_head (output.weight) and token embedding tensors.

The goal: keep native NVFP4 density across the whole model for Blackwell, and offer a size/precision ladder for the tensors that most affect output quality and decode speed.

[!IMPORTANT] Read this before downloading. These files need a Muse-Glimmer NVFP4 fix that is not merged upstream yet (ggml-org/llama.cpp#27178). On a stock llama.cpp build they load but generate garbage (repetition artifacts, wrong-layout attention).

Verified path: gabrielcosi's llama.cpp fork, commit 9ec0ea18 or later. That fork contains the complete runtime + converter fix, and is the only thing we actually tested with. Our sanity checks ran on that fork (identical results on our own build of the same fix).

Other GGUF-consuming inference engines might handle the interleaved Q/K layout correctly without the fork, but we have not tested any of them. Treat gabrielcosi's fork as the only verified runtime until #27178 lands upstream.

The model is a native VLM (images and video), but the vision tower is always BF16 in the source checkpoint and is not included in these files (no mmproj yet). We have not tested vision with these conversions; treat them as text-only for now.

Follow along & support

I post updates on new conversions, benchmarks, and what I'm working on over on my site. Follow along there to keep up with new releases and the work in progress. If you'd like to support more of it, you will find the options there. I do this on consumer hardware and like seeing how far it goes. More is on the way.

🌐 esatapedico.com. Updates, work-in-progress, and ways to support.

The seven files

Sizes below are the on-disk file sizes in decimal gigabytes (1 GB = 1,000,000,000 bytes), matching what Hugging Face's file browser shows for each file. Earlier versions of this card reported binary values (GiB) while labeling them "GB", which made the files appear ~1 GB smaller than the actual sizes — that has been corrected.

Tensor layout

All seven tiers share the same structure:

The NVFP4 backbone is the same set of bytes in every tier (verified: per-tensor SHA-256). The only difference between tiers is the quantization of the two head tensors.

Why only lm_head and token_embd vary

The source checkpoint stores attention projections as native NVFP4 (GGML type 40). llama-quantize's override-file mechanism can change the quantization of non-NVFP4 tensors (like the BF16 lm_head and token embedding) but it cannot requantize from NVFP4 to another format. This means the 416 attention + MLP tensors in each transformer block are locked at their source precision.

This differs from Qwen3.8-27B, whose attention projections start as BF16 in the source checkpoint and can be freely requantized tier-by-tier (Q8_0 for most layers, BF16 for the final layer). Muse-Glimmer's NVFP4 source doesn't permit that path, and the quality case for promoting it is weaker anyway:

• Sliding window attention. Muse-Glimmer attends locally (window of 2048) in 3 out of every 4 layers. Quantization artifacts in local attention patterns don't compound across the full context the way they do in Qwen3.8's global attention, so the marginal value of higher-precision attention weights is reduced.
• NVFP4 is already well-calibrated. Per-tensor scaling in NVFP4 compensates for the narrow dynamic range, making the backbone's quantization noise floor low relative to the attention pattern itself. The dominant quality lever is the vocabulary interface (lm_head + token_embd), not the internal projections.

Tiering those two head tensors gives a meaningful quality gradient while keeping the backbone density at native NVFP4.

QK-norm absorption

The source checkpoint applies a scaleless RMSNorm followed by qk_scale_factor=3.87 on the Q/K projections (per MuseGlimmer's implementation). At conversion time this is synthesized into two dedicated tensors per layer: attn_q_norm.weight (a 128-dim vector filled with the constant 3.87) and attn_k_norm.weight (a 128-dim vector of ones), which llama.cpp's runtime multiplies into Q and K after RMS norm. This keeps the math equivalent to the source's QK-normed output without requiring a runtime scale-factor lookup. These 104 tensors are part of the F32 set shared byte-identically across all tiers.

Sliding window attention

Muse-Glimmer uses a sliding window attention pattern: [Local, Local, Local, Global] with period 4. Three out of every four layers attend only within a local window; the fourth layer attends globally. This is handled natively by llama.cpp's GGUF metadata.

Attribution & provenance

This is a derivative work built entirely from existing Apache-2.0 artifacts. Nothing here was trained or fine-tuned. Credit belongs to:

1. Meta's Superintelligence Lab for the base model, meta-models/Muse-Glimmer-30B (Apache-2.0): 30B dense VLM, 52 layers, 6656 hidden, 32 heads, 2 KV heads, 128 head_dim, sliding window attention, 131K native context.
2. Red Hat AI for the NVFP4-quantized checkpoint, RedHatAI/Muse-Glimmer-30B-NVFP4 (Apache-2.0), which is the direct source for this conversion.
3. llama.cpp contributors for the GGUF format, NVFP4 (GGML type 40) support, and the conversion tooling.
4. Gabriel Cosi (gabrielcosi, gabrielcosi.dev) for diagnosing and fixing the NVFP4 conversion/runtime bugs: scale wiring in muse-glimmer.cpp, the QK-norm synthesis trigger, and the modify_nvfp4_tensor hook that unpermutes Q/K back to the interleaved RoPE layout before repacking (commit 9ec0ea18 in his llama.cpp fork, tracking ggml-org/llama.cpp#27178). His work is the reason these files generate coherent output at all.
5. This repo's author for the GGUF conversion, QK-norm synthesis, the tier splicing, and the verification.

The NVFP4 tensors are native GGML type 40, preserved from the source checkpoint through GGUF conversion with no re-quantization round trip.

How this was made

1. RedHatAI/Muse-Glimmer-30B-NVFP4 (safetensors, NVFP4) was converted to GGUF with convert_hf_to_gguf.py --outtype auto, synthesizing the QK-norm scale factor into per-layer attn_q_norm/attn_k_norm vectors at conversion time and, critically, unpermuting Q/K back to the interleaved (NORM) RoPE layout via the modify_nvfp4_tensor hook, which NVFP4-packed tensors need because they bypass modify_tensors (gabrielcosi's fix). This yields a base GGUF (1,563 tensors) with BF16 lm_head and token embedding that generates coherent output.
2. Each tier produced with llama-quantize --tensor-type-file <overrides> from the base parent. The per-tier override maps are in this repo (overrides-very-low.txt through overrides-highest.txt).

The NVFP4 backbone is byte-identical across the family: the 416-tensor backbone of all seven tiers is the same set of bytes in every tier (verified: per-tensor SHA-256).

Naive quality checks (not a benchmark)

Take these as rough sanity observations, not a benchmark; they are not a proper evaluation. Method: llama-perplexity on a small mixed-source text (source-diverse.txt, ~2.6k tokens, 6 chunks × 512 ctx) plus a single llama-cli generation (one prompt, one seed, 220 tokens), all on the gabrielcosi fork. No task suites, no long-context stress, no vision testing, no latency/throughput measurements, no comparison against the reference implementation.

With that said, all tiers generated coherent reasoning and answers with no repetition artifacts, and perplexity is in a healthy band:

MID-HIGH (Q8_0 lm_head + Q8_0 token_embd) was added after this measurement pass. It shares the same byte-identical NVFP4 backbone as every tier above (verified per-tensor), and its head tensors are the same precision or higher than MEDIUM's (Q8_0/Q6_K → Q8_0/Q8_0), so its perplexity/speed sit at or slightly better than MEDIUM's. It generated coherent output with no repetition artifacts in a load test, but was not given its own PPL/speed run in this pass.

For reference, a Q4_K_M quantized control (non-NVFP4, standard quantize path) measured 7.88 ± 0.54 under the same method; the ~4% gap is consistent with quantization noise. The important observation is that the NVFP4 family sits in the same quality band as a normal K-quant, rather than the ~70× worse perplexity the pre-fix conversion produced.

Naive speed observations

Same caveat as above: single-machine, single-run numbers from llama-bench on 2× NVIDIA Blackwell 16 GB (RTX 5070 Ti + RTX 5060 Ti), -p 512 -n 128 -r 3, all layers on GPU.

Context matters: these were measured at short context (512-token prompt, 128 generated tokens). They are not 131K-context numbers. Prefill t/s in particular degrades as the prompt grows, so do not compare these against numbers from longer-context runs or different hardware.

Prefill is roughly 2500-2600 t/s across the family, and slightly higher than the KQuant control; generation tracks the lm_head precision (33 t/s for the Q3_K-head tier down to ~28.5 t/s for the BF16-head tiers). The KQuant control sits at 30.6 t/s, in the same band.

DFlash speculative decoding

Muse Glimmer ships with a DFlash drafter (a small block-diffusion model that proposes 16-token blocks for the main model to verify in parallel). The official quantized drafter is published by Meta at meta-models/Muse-Glimmer-30B-GGUF (dflash-Muse-Glimmer-30B-Q4_K_M.gguf, ~1.5 GB, Apache-2.0). It is a separate model, so it works with any Muse Glimmer main model, including these NVFP4 tiers.

Quick check on the HIGHEST tier, short context (single run, llama-cli, 200 tokens): 27.9 t/s without the drafter, 43.1 t/s with it (~1.5x). Same context caveat as the speed table above.

llama-server \
  --model Muse-Glimmer-30B-NVFP4-MEDIUM.gguf \
  -md dflash-Muse-Glimmer-30B-Q4_K_M.gguf -ngld 99 \
  --flash-attn on --n-gpu-layers 99

Same runtime caveat as everything else: this needs the unmerged #27178 fix (gabrielcosi's fork). A [spec] failed to measure draft model memory warning at startup is harmless.

Vision (mmproj)

Meta also publishes a perception encoder for image input: mmproj-Muse-Glimmer-30B-Q4_K_M.gguf (~1.3 GB, Apache-2.0) in the same repo. It loads cleanly with our tiers, but we have not tested image input end to end; treat vision as untested.

Repository contents

• Muse-Glimmer-30B-NVFP4-VERY-LOW.gguf (15.19 GB)
• Muse-Glimmer-30B-NVFP4-LOW.gguf (15.85 GB)
• Muse-Glimmer-30B-NVFP4-MEDIUM.gguf (16.71 GB)
• Muse-Glimmer-30B-NVFP4-MID-HIGH.gguf (17.03 GB)
• Muse-Glimmer-30B-NVFP4-HIGH.gguf (17.97 GB)
• Muse-Glimmer-30B-NVFP4-VERY-HIGH.gguf (18.29 GB)
• Muse-Glimmer-30B-NVFP4-HIGHEST.gguf (19.55 GB)
• overrides-very-low.txt through overrides-highest.txt: per-tensor quantization maps for reproduction (includes overrides-mid-high.txt)

The DFlash drafter and mmproj files are not in this repo; they are Meta's official companions, published in meta-models/Muse-Glimmer-30B-GGUF. Download them from there and pass them with -md / --mmproj as shown above.

SHA-256

cf253478d919459a3551f33b1632bef88369bf57ef7054dc67d46c09f184e7b5  Muse-Glimmer-30B-NVFP4-VERY-LOW.gguf
f28ac88a6c6d5f003e66b7e08f7cf2238b4a5ed5abc84bb7a29c3f44dbe14c22  Muse-Glimmer-30B-NVFP4-LOW.gguf
3962383a2f4e3a026ae786c8edabdab456de888bb8b5c8138e316e19bf1afd2f  Muse-Glimmer-30B-NVFP4-MEDIUM.gguf
6e10d73630bd30456e74a1781f0eb2c8de49317a30821828da054b65d62d3e21  Muse-Glimmer-30B-NVFP4-MID-HIGH.gguf
882bbd9d69493f00631470e8b2387d5497e636e4b9cb1d6b2c936ba6dd1095d2  Muse-Glimmer-30B-NVFP4-HIGH.gguf
3a96f812392c1a31b89a07b7cf219e87593d9015554f2a204321cca027446bb2  Muse-Glimmer-30B-NVFP4-VERY-HIGH.gguf
338bc19c8e6b70b5717f7e4541b53c439198b383cc45b75d3f6dbf786e384e6e  Muse-Glimmer-30B-NVFP4-HIGHEST.gguf

Usage

llama.cpp / llama-server

llama-server \
  --model Muse-Glimmer-30B-NVFP4-MEDIUM.gguf \
  --ctx-size 131072 \
  --flash-attn on \
  --temp 0.7 --top-p 0.95 --top-k 40 \
  --repeat-penalty 1.05

• Requires a recent llama.cpp with NVFP4 (GGML type 40) CUDA kernels and sm_120 support (Blackwell) plus the unmerged #27178 fix, i.e. gabrielcosi's fork for now (see the important note at the top).
• No MTP speculative decoding; this model does not have an MTP head.
• For vision input, pass --mmproj once a mmproj file is available; these files do not include one.
• Muse-Glimmer's recommended sampling: temp 0.7 / top_p 0.95 / top_k 40. A --repeat-penalty ≥ 1.05 can help on long generations, but the severe word-repetition ("short short short…") seen in earlier conversions was a symptom of the Q/K layout bug, not a sampling requirement; with the fix, the model generates cleanly under default sampling.

Notes

• Hardware: tested on 2× NVIDIA Blackwell 16 GB (RTX 5070 Ti + RTX 5060 Ti). All tiers fit a single 16 GB card at reasonable context; the larger tiers prefer a split or smaller context.
• Serving caveat: the model's Onyx-style chat template emits <|start|>assistant to=user<|message|> (recipient syntax). Some llama.cpp-based servers (e.g. certain LocalAI backend builds) reject this in their chat-output PEG parser and return an error or fallback text. If you hit that, use a recent llama.cpp llama-server directly (raw completion works; the PEG parser limitation is chat-path only) or a template override.
• The vision tower is always BF16 in the source and was not included in the NVFP4 quantization. These files are text-only; a separate mmproj file would be needed for image/video input.
• Non-NVFP4 (K-quant) GGUFs of the base model may be available in the upstream repositories.

License

Apache-2.0, identical to every upstream artifact. The base model license governs; GGUF conversion and quantization are transformations, not new training. When redistributing, please retain attribution to Meta (base model) and Red Hat AI (NVFP4 quantization) as above.

"Muse Glimmer" is the name of a model released by Meta's Superintelligence Lab. This repository is not affiliated with, sponsored by, or endorsed by Meta or Red Hat.

Note on this card

This model card was written by an AI assistant at the request of the repository author, who did the engineering. As with any AI-generated text, there may be errors; please verify anything important (hashes, sizes, commands) against the file itself before relying on it.