Muse-Glimmer-30B-Hermes-Agentic-NVFP4

NVFP4 (compressed-tensors / nvfp4-pack-quantized) export of the Hermes-agentic Muse Glimmer 30B student.

• BF16 / FP16 merge: vcruz305/Muse-Glimmer-30B-Hermes-Agentic
• GGUF ladder: vcruz305/Muse-Glimmer-30B-Hermes-Agentic-GGUF
• Serve recipes (SGLang + vLLM on DGX Spark SM121): github.com/vcruz305/muse-glimmer-nvfp4-spark-serve
• Base: meta-models/Muse-Glimmer-30B (Apache-2.0)

Format

• quantization_config.format: nvfp4-pack-quantized
• quant_method: compressed-tensors
• ~22 GB on disk (2 safetensor shards)
• Vision / projector / lm_head / linear-attn / MTP paths ignored in the NVFP4 recipe (see recipe.yaml)

Serving (preferred: SGLang on Spark)

hf download vcruz305/Muse-Glimmer-30B-Hermes-Agentic-NVFP4 \
  --local-dir $HOME/models/Muse-Glimmer-30B-Hermes-Agentic-NVFP4
export MODEL=$HOME/models/Muse-Glimmer-30B-Hermes-Agentic-NVFP4
# from https://github.com/vcruz305/muse-glimmer-nvfp4-spark-serve
bash scripts/launch_sglang_nvfp4_spark.sh

Critical SM121 flags (do not leave on auto):

• --fp4-gemm-backend flashinfer_cutlass
• --cuda-graph-backend-decode full
• --cuda-graph-backend-prefill disabled
• --reasoning-parser muse + --tool-call-parser muse
• --language-model-only

vLLM secondary path is in the same recipe repo (scripts/launch_vllm_nvfp4.sh).

Evaluation (sixcat, SGLang, thinking on)

Host: DGX Spark GB10 · endpoint OpenAI-compatible · policy strict · thinking on · limit 20 · sixcat ≥0.4.4 · full 120 after --retry remaining.

Suite throughput ≈ 11.9 tok/s on that run. Instruct/code still carried length/loop flags.

Model Details

Hermes-agentic SFT of Muse Glimmer so the model calls one or two tools and stops. Vision weights remain in the tree but were frozen for the text/tools student. This repo is the NVFP4 deployable of that merge — not a second SFT.

• Developed by: Victor Cruz (vcruz305)
• License: Apache-2.0
• Quant scheme: NVFP4 via compressed-tensors (recipe.yaml in-repo)

How to Get Started

1. Download this repo.
2. Follow muse-glimmer-nvfp4-spark-serve.
3. Hit http://127.0.0.1:8201/v1 with model id muse-glimmer-30b-nvfp4.