Muse-Glimmer-30B-lecore-mlx
MLX mixed-precision quantization of meta-models/Muse-Glimmer-30B, with the bit width of every
linear layer picked by leCore Unicron requantize (per-tensor bits chosen by measurement), ported to GPU for this model size.
How the bits were chosen
- Each decoder linear tensor was quantized alone (MLX affine, group 64 during measurement) at 2, 3 and 4 bits; the damage was measured as KL(fp16 || quantized) on next-token distributions over a calibration mix (3066 WikiText-2 train tokens, 1022 Python-source tokens, and leCore's structured probe: prose, code, SQL, markdown).
- Bits were assigned greedily: start at 4 bits everywhere, repeatedly take the downgrade with the least measured damage per byte saved until the predicted file size fit the budget. Embeddings 4-bit, lm_head 6-bit, group size 64.
- Converted with
mlx_lm.convertand a per-layerquant_predicate. Bit histogram (linear tensors): {'2': 0, '3': 0, '4': 416, '8': 0, '16': 0}.
Measured results (held-out: WikiText-2 test + a different Python-source slice, seq 512)
Perplexity was computed on the exact shipped weights (MLX tensors dequantized with mx.dequantize, run in bf16 on GPU).
"bpw" = integer bits + 0.5 for the bf16 scale/bias per group of 64.
| Model | Size (GB, weights) | bpw | WikiText-2 ppl (32704 tok) | Python-code ppl (8176 tok) |
|---|---|---|---|---|
| bf16 reference | – | 16 | 8.2001 | 3.2983 |
| this repo (leCore Unicron mix) | 16.01 | 4.597 | 8.4699 | 3.4615 |
| stock MLX 3-bit, group 64 | 12.19 | 3.500 | 9.8255 | 4.1635 |
Benchmarks (lm-eval-harness 0.4.9, run on the shipped weights dequantized to bf16 on GPU)
MMLU 5-shot on the first 10 questions of each of the 57 subjects; GSM8K 5-shot on the first 100 test problems (greedy, 320 new tokens, no chat template). These are subsets, so expect about ±2–4 points of sampling noise.
| Model | MMLU (5-shot) | GSM8K strict | GSM8K flexible |
|---|---|---|---|
| leCore mix (this repo) | 81.8% | 96.0% | 96.0% |
| stock 3-bit affine, group 64 (simulated) | 78.6% | 90.0% | 90.0% |
Only measured numbers are reported. No downstream-task benchmarks were run. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.
Files in this repo
| file | size | what it is |
|---|---|---|
model-00001-of-00003.safetensors |
5.31 GB | quantized weights |
model-00002-of-00003.safetensors |
5.37 GB | quantized weights |
model-00003-of-00003.safetensors |
5.33 GB | quantized weights |
tokenizer.json |
28.1 MB | tokenizer |
model.safetensors.index.json |
127.1 KB | config / tokenizer |
config.json |
94.4 KB | model config |
tokenizer_config.json |
80.0 KB | config / tokenizer |
chat_template.jinja |
10.0 KB | chat template |
generation_config.json |
202 B | generation defaults |
Usage (Apple Silicon)
pip install -U mlx-lm
mlx_lm.generate --model staccs/Muse-Glimmer-30B-lecore-mlx --prompt "Hello" --max-tokens 256 --max-kv-size 8192
Weights are ~16.01 GB. On a 36 GB Mac you may need to raise the GPU wired-memory limit, e.g. sudo sysctl iogpu.wired_limit_mb=30000.
License
Base model: meta-models/Muse-Glimmer-30B, released under the Apache License 2.0. Only quantization was done here, with no retraining. Vision: this MLX export is text-only, because mlx-lm's Muse-Glimmer port has no vision tower. The companion GGUF repo includes an unquantized f16 mmproj vision projector for llama.cpp image input, if its conversion succeeded. Perplexity and benchmarks below are text-only.
Related builds
Same base model, other formats in the lecore set:
| repo | format | size |
|---|---|---|
| Muse-Glimmer-30B-lecore-GGUF | GGUF (llama.cpp) | 36.63 GB |