Muse-Glimmer-30B-W8A8-Int8

An inference-oriented W8A8 quantization of Muse-Glimmer-30B, using 8-bit weights and dynamic 8-bit activations. This release is designed to retain the behavior of the original BF16 model while substantially reducing the memory bandwidth, storage costs of the language model and achieve the best speeds possible on dual RTX 3090s. Weight-only comparisons are already very close to the BF16 reference. A broader activation-side validation campaign is still in progress; see Validation status for the exact status of the current claims.

[!IMPORTANT] This is an early validation release. The weight quantization itself closely matches the BF16 model, but the full effect of INT8 activations is still being characterized across layers, sequence lengths, workloads, and generation regimes. Results will be added to this model card as they become available.

Model summary

The language-model linear layers are quantized. Components that are especially sensitive or not part of the main W8A8 execution path—such as embeddings, the language-model head, and non-language components—remain outside the INT8 quantization scope.

Why this quantization exists

The objective of this release is not merely to reduce the checkpoint size. It is to provide a practical W8A8 version of Muse-Glimmer-30B that can benefit from optimized INT8 kernels while preserving the model's BF16 behavior as closely as possible.

The quantization process therefore prioritizes:

• low divergence from the BF16 reference
• stable behavior across individual layers, layer groups, and the complete model
• preservation of instruction following, coding, tool use, and long-context behavior
• high-throughput inference on hardware with efficient INT8 tensor operations
• reproducible, workload-aware validation rather than relying on a single aggregate similarity metric

Validation status

Validation is deliberately split into two separate questions:

1. Weight quantization fidelity: how closely the dequantized INT8 weights reproduce the BF16 weights and their weight-only outputs.
2. End-to-end W8A8 fidelity: how dynamic INT8 activations affect hidden states, logits, token decisions, generation trajectories, and downstream capabilities.

The first is already very close to the BF16 reference. The second is still being evaluated comprehensively and should not be inferred from weight-only measurements.

Weight quantization

BF16 vs W8A8 validation results

Static audits of the INT8 weights, quantization scales, saturation behavior, and corrected LogEq effective weights were all successful. They indicate that the checkpoint itself is healthy and remains very close to the BF16 reference.

Dynamic INT8 activation validation

The following section is intentionally reserved for the full activation evaluations. The final report will cover individual layers, architectural groups, and complete end-to-end behavior.

Usage

vLLM server

docker run --rm \
  --name muse-glimmer-w8a8 \
  --gpus all \
  --ipc=host \
  -p 8080:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  AlexanderKyng/Muse-Glimmer-30B-W8A8-Int8 \
  --generation-config vllm \
  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":64}' \
  --tensor-parallel-size 2 \
  --max-num-seqs 5 \
  --language-model-only \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Docker might not be strictly necessary but as of today, I only tried using this command, not vLLM directly.

OpenAI-compatible client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local",
)

response = client.chat.completions.create(
    model="AlexanderKyng/Muse-Glimmer-30B-W8A8-Int8",
    messages=[
        {
            "role": "user",
            "content": "Write a Python function that merges overlapping intervals.",
        }
    ],
    temperature=1.0,
)

print(response.choices[0].message.content)

Hardware notes

W8A8 speedups depend on the inference backend, GPU architecture, available INT8 kernels, batch size, sequence length, and tensor-parallel configuration. It was 100% developed with the Ampere GPUs (and espacially the RTX 3090s) and RTX 4090s in mind.

Intended use

This model is intended for users who want to evaluate or deploy Muse-Glimmer-30B with a lower-bandwidth W8A8 execution path, especially for:

• local inference
• coding and agentic workloads
• long-context experiments
• throughput, memory, and quantization research

It is also suitable as a reproducible target for studying activation quantization and layerwise divergence in modern large language models.

Limitations

• Full activation-side equivalence with BF16 is not yet proven.
• Small numerical errors can accumulate across layers and alter generation trajectories, even when aggregate similarity metrics appear excellent.
• Stochastic generations are not expected to match BF16 token-for-token; distributional comparisons require repeated sampling and controlled seeds.
• Backend support and realized performance vary by hardware and software version.
• This quantization does not change the underlying model's knowledge, safety characteristics, biases, or license obligations.

Reproducibility

The final quantization and validation release will document:

• the exact quantization configuration
• calibration data composition and preprocessing
• software and kernel versions
• deterministic weight-level checks
• layerwise activation captures
• end-to-end BF16 versus W8A8 comparisons
• throughput and memory measurements

Acknowledgements

Thanks to the Meta, the creators of Muse-Glimmer-30B and to the open-source communities behind vLLM, compressed-tensors, Transformers, and the wider quantization ecosystem.

Citation

If this quantized model is useful in your work, please cite the original Muse-Glimmer-30B model and link to this repository.

@misc{muse_glimmer_30b_w8a8_logeq_rtn_v1,
  title        = {Muse-Glimmer-30B-W8A8-Int8},
  author       = {AlexanderKyng},
  year         = {2026},
  howpublished = {Hugging Face model repository}
}