gemma-3-27b-it-lecore-mlx
MLX mixed-precision quantization of google/gemma-3-27b-it, with the bit width of every
linear layer picked by leCore Unicron requantize (per-tensor bits chosen by measurement), ported to GPU for this model size.
How the bits were chosen
- Each decoder linear tensor was quantized alone (MLX affine, group 64 during measurement) at 2, 3 and 4 bits; the damage was measured as KL(fp16 || quantized) on next-token distributions over a calibration mix (3066 WikiText-2 train tokens, 1022 Python-source tokens, and leCore's structured probe: prose, code, SQL, markdown).
- Bits were assigned greedily: start at 4 bits everywhere, repeatedly take the downgrade with the least measured damage per byte saved until the predicted file size fit the budget. Embeddings 4-bit, lm_head 6-bit, group size 64.
- Converted with
mlx_lm.convertand a per-layerquant_predicate. Bit histogram (linear tensors): {'2': 57, '3': 271, '4': 106, '8': 0}.
Measured results (held-out: WikiText-2 test + a different Python-source slice, seq 512)
Perplexity was computed on the exact shipped weights (MLX tensors dequantized with mx.dequantize, run in bf16 on GPU).
"bpw" = integer bits + 0.5 for the bf16 scale/bias per group of 64.
| Model | Size (GB, weights) | bpw | WikiText-2 ppl (32704 tok) | Python-code ppl (8176 tok) |
|---|---|---|---|---|
| bf16 reference | – | 16 | 10.7867 | 4.9757 |
| this repo (leCore Unicron mix) | 11.79 | 3.491 | 12.5122 | 5.4731 |
| stock MLX 3-bit, group 64 | 11.82 | 3.500 | 13.1617 | 5.5558 |
Benchmarks (lm-eval-harness 0.4.9, run on the shipped weights dequantized to bf16 on GPU)
MMLU 5-shot on the first 10 questions of each of the 57 subjects; GSM8K 5-shot on the first 100 test problems (greedy, 320 new tokens, no chat template). These are subsets, so expect about ±2–4 points of sampling noise.
| Model | MMLU (5-shot) | GSM8K strict | GSM8K flexible |
|---|---|---|---|
| leCore mix (this repo) | 75.1% | 89.0% | 89.0% |
| stock 3-bit affine, group 64 (simulated) | 75.3% | 68.0% | 74.0% |
Only measured numbers are reported. No downstream-task benchmarks were run. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.
Files in this repo
| file | size | what it is |
|---|---|---|
model-00001-of-00003.safetensors |
5.36 GB | quantized weights |
model-00002-of-00003.safetensors |
5.36 GB | quantized weights |
model-00003-of-00003.safetensors |
1.06 GB | quantized weights |
tokenizer.json |
33.4 MB | tokenizer |
tokenizer.model |
4.7 MB | support file |
tokenizer_config.json |
1.2 MB | config / tokenizer |
model.safetensors.index.json |
145.3 KB | config / tokenizer |
config.json |
94.5 KB | model config |
chat_template.jinja |
1.5 KB | chat template |
special_tokens_map.json |
670 B | config / tokenizer |
generation_config.json |
210 B | generation defaults |
Usage (Apple Silicon)
pip install -U mlx-lm
mlx_lm.generate --model staccs/gemma-3-27b-it-lecore-mlx --prompt "Hello" --max-tokens 256 --max-kv-size 8192
Weights are ~11.79 GB. On a 36 GB Mac you may need to raise the GPU wired-memory limit, e.g. sudo sysctl iogpu.wired_limit_mb=30000.
License
Base model: google/gemma-3-27b-it (weights taken from the identical public mirror unsloth/gemma-3-27b-it). Gemma is provided under and subject to the Gemma Terms of Use (https://ai.google.dev/gemma/terms) and its Prohibited Use Policy (https://ai.google.dev/gemma/prohibited_use_policy). This export is text-only: the vision tower was dropped. Only quantization was done here: no retraining.
Related builds
Same base model, other formats in the lecore set:
| repo | format | size |
|---|---|---|
| gemma-3-27b-it-lecore-GGUF | GGUF (llama.cpp) | 28.23 GB |
| gemma-3-27b-it-lecore-hrr | gated-HRR gates | 29.7 KB |