gpt-oss-120b-tq3

TurboQuant 3-bit quantized version of openai/gpt-oss-120b using TurboQuant-MLX.

Documentation: manjunathshiva.github.io/turboquant-mlx: the GPT-OSS page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents and memory and macOS settings.

Model Details

• Base Model: openai/gpt-oss-120b (MoE: 120B params)
• Quantization: TurboQuant 3-bit (Hadamard rotation + Lloyd-Max codebook)
• Parameters: ~120B
• Size: ~48 GB (vs ~240 GB BF16)
• Group Size: 64

Will it fit my Mac?

Don't guess, and don't download 51.4 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/gpt-oss-120b-tq3

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      51.4 GB
  KV cache             36.6 KB/token  (hybrid: 18/36 full-attention, 18 sliding (window 128))

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/gpt-oss-120b-tq3 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Requirements

# macOS with Apple Silicon (M1/M2/M3/M4/M5)
pip install turboquant-mlx-full 

Quick Start

Generate text

python -m turboquant_mlx.generate \
    --model ~/path/to/gpt-oss-120b-tq3 \
    --prompt "Why is the sky blue? Explain in simple terms." \
    --max-tokens 200 \
    --temp 0.7

With KV Cache Compression

For longer prompts/context, use KV cache compression:

python -m turboquant_mlx.demo_kv \
    --model ~/path/to/gpt-oss-120b-tq3 \
    --prompt "Your long prompt here..." \
    --max-tokens 200 \
    --tq-bits 3

Results

With KV cache compression:

How It Works

TurboQuant applies:

1. Hadamard rotation - random +/- 1 scaling to decorrelate weights
2. Lloyd-Max codebook - optimal scalar quantization via k-means
3. Group-wise scaling - per-group float16 scales for precision

This achieves much better quality than standard affine quantization at the same bit-width. On 100B+ models, the 4x smaller KV cache reduces memory bandwidth more than dequant adds — generation is faster than FP16 baseline!

License

Apache 2.0 (same as base model)

Citation

@article{zandieh2025turboquant,
  title={TurboQuant: A Unified Framework for Extremely Low-Bit Weight and KV Cache Quantization},
  author={Zandieh, Amir and Han, Minsik and Dalca, Andre and Shin, Jungwoo and Wang, Brian and Zhang, Yichao and Bordegoni, Matteo and Tian, Yuan and others},
  year={2025}
}