Laghu-MLX-4bit (unofficial community 4-bit conversion)
This is a 4-bit MLX quantization of udgamlabs/Laghu-MLX, the MLX build of udgamlabs/Laghu, a 152M-parameter small language model trained by Udgam Labs. All credit for the model goes to Udgam Labs. This repo only changes the weight precision.
[!WARNING] Quality caveat. This is an unofficial conversion, not released by Udgam Labs. Udgam Labs deliberately chose not to publish a 4-bit MLX build because it loses quality clearly. They report mean KL 0.337 and top-1 agreement 69.9% against their F16 reference. Their bf16 MLX build scores KL 0.0009 and 98.0%. I did not run quality benchmarks on this 4-bit copy. For quality, use the bf16 udgamlabs/Laghu-MLX. This copy is here for speed and memory experiments.
Quantization
- Tool:
mlx_lm.convert -q --q-bits 4 --q-group-size 64(affine, group size 64) - Effective size: 4.506 bits per weight (model.safetensors is about 86 MB, vs 303.6 MB for upstream bf16)
- Versions: mlx-lm 0.32.2, mlx 0.32.4, Python 3.12
- Architecture:
gemma4_text, max context 16,384 tokens, 512-token sliding window - The tokenizer, chat template and generation config are unchanged from upstream
Usage
pip install -U mlx-lm
mlx_lm.generate --model pratikgx/Laghu-MLX-4bit --prompt "Who are you?" --max-tokens 256
from mlx_lm import load, generate
model, tokenizer = load("pratikgx/Laghu-MLX-4bit")
messages = [{"role": "user", "content": "Write a haiku about autumn rain."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))
The chat template supports an enable_thinking switch. Upstream recommends temperature 0.7, top-p 0.95 and top-k 64,
and keeping context at 16,384 tokens or less.
Speed (bf16 vs this 4-bit copy)
Measured on an Apple M4 Pro (16 GPU cores, 24 GB) running macOS 27.0.1, with mlx 0.32.4 and mlx-lm 0.32.2. Settings: greedy decoding, prompts taken from Pride and Prejudice, 256 generated tokens (ignore EOS), prefill step 2048, 1 warmup run, then the median of 3 runs.
| Prompt tokens | bf16 decode | 4-bit decode | bf16 prefill | 4-bit prefill | bf16 peak mem | 4-bit peak mem |
|---|---|---|---|---|---|---|
| 512 | 364 tok/s | 402 tok/s | 11,516 tok/s | 11,844 tok/s | 0.66 GB | 0.38 GB |
| 2,048 | 356 tok/s | 409 tok/s | 11,264 tok/s | 10,825 tok/s | 1.17 GB | 1.11 GB |
| 8,192 | 328 tok/s | 397 tok/s | 9,198 tok/s | 8,876 tok/s | 1.23 GB | 1.15 GB |
| 16,000 | 303 tok/s | 379 tok/s | 7,554 tok/s | 7,069 tok/s | 1.26 GB | 1.15 GB |
4-bit decodes about 10–25% faster. Prefill speed is about the same. Peak memory is lower only at short prompts, because the KV cache dominates at longer contexts.
The quality benchmarks (GSM8K, IFEval, ARC-Easy, HellaSwag) were run on the bf16 model only, not on this 4-bit copy. Raw results, settings and the harness are in the dataset pratikgx/launch-bench-results.
License and attribution
Apache-2.0, the same as upstream. See NOTICE, copied unchanged from udgamlabs/Laghu-MLX: the weights and tokenizer are
© 2026 Udgam Labs LLP, and the chat template is Google's Gemma 4 template (Apache-2.0).
Write-up on GitHub
How I ran this, what broke, and the scripts: Laghu 152M: speed and quality. It's part of Local AI field notes, where I log every model I run on my Mac mini.