Seed-OSS-36B-Instruct-lecore-mlx
MLX mixed-precision quantization of ByteDance-Seed/Seed-OSS-36B-Instruct, with the bit width of every
linear layer picked by leCore Unicron requantize (per-tensor bits chosen by measurement), ported to GPU for this model size.
How the bits were chosen
- Each decoder linear tensor was quantized alone (MLX affine, group 64 during measurement) at 2, 3 and 4 bits; the damage was measured as KL(fp16 || quantized) on next-token distributions over a calibration mix (3066 WikiText-2 train tokens, 1022 Python-source tokens, and leCore's structured probe: prose, code, SQL, markdown).
- Bits were assigned greedily: start at 4 bits everywhere, repeatedly take the downgrade with the least measured damage per byte saved until the predicted file size fit the budget. Embeddings 4-bit, lm_head 6-bit, group size 64.
- Converted with
mlx_lm.convertand a per-layerquant_predicate. Bit histogram (linear tensors): {'2': 8, '3': 116, '4': 324, '8': 0, '16': 0}.
Measured results (held-out: WikiText-2 test + a different Python-source slice, seq 512)
Perplexity was computed on the exact shipped weights (MLX tensors dequantized with mx.dequantize, run in bf16 on GPU).
"bpw" = integer bits + 0.5 for the bf16 scale/bias per group of 64.
| Model | Size (GB, weights) | bpw | WikiText-2 ppl (32704 tok) | Python-code ppl (8176 tok) |
|---|---|---|---|---|
| bf16 reference | – | 16 | 11.2952 | 4.4792 |
| this repo (leCore Unicron mix) | 18.99 | 4.201 | 10.8602 | 4.5305 |
| stock MLX 3-bit, group 64 | 15.82 | 3.500 | 12.1364 | 4.8896 |
Only measured numbers are reported. No downstream-task benchmarks were run. Short-context perplexity does not tell you everything about quality, so check outputs for your own use case.
Files in this repo
| file | size | what it is |
|---|---|---|
model-00001-of-00004.safetensors … (4 safetensors shards) |
18.99 GB total | quantized weights, sharded |
tokenizer.json |
11.9 MB | tokenizer |
model.safetensors.index.json |
143.1 KB | config / tokenizer |
config.json |
97.5 KB | model config |
tokenizer_config.json |
23.8 KB | config / tokenizer |
chat_template.jinja |
7.7 KB | chat template |
special_tokens_map.json |
432 B | config / tokenizer |
generation_config.json |
172 B | generation defaults |
Usage (Apple Silicon)
pip install -U mlx-lm
mlx_lm.generate --model staccs/Seed-OSS-36B-Instruct-lecore-mlx --prompt "Hello" --max-tokens 256 --max-kv-size 8192
Weights are ~18.99 GB. On a 36 GB Mac you may need to raise the GPU wired-memory limit, e.g. sudo sysctl iogpu.wired_limit_mb=30000.
License
Base model: ByteDance-Seed/Seed-OSS-36B-Instruct, released by ByteDance Seed under the Apache License 2.0. Only quantization was done here: no retraining.