Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32

Hybrid TurboQuant quantization of Qwen/Qwen3-235B-A22B-Instruct-2507 — 3-bit attention, 2-bit experts (group size 32) — produced with TurboQuant-MLX.

A 235B-parameter MoE compressed from ~470 GB (BF16) to 70.51 GB, small enough to stream on a 16 GB Mac and run fully resident on a 96 GB+ Apple Silicon machine.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Qwen3-235B-A22B page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, models bigger than RAM and memory and macOS settings.

Model Details

• Base model: Qwen/Qwen3-235B-A22B-Instruct-2507 — Mixture-of-Experts (qwen3_moe)
• Architecture: 94 layers, hidden size 4096, 128 routed experts (top-8), MoE intermediate 1536, ~235B total / ~22B active params
• Quantization: Hybrid TurboQuant (Hadamard rotation + Lloyd-Max codebook), group size 32
  • Attention → 3-bit (q/k/v/o_proj + lm_head): 377 Linear layers
  • Experts → 2-bit (gate/up/down_proj of every expert): 282 SwitchLinear layers
  • Routers → full precision (mlp.gate is auto-skipped — never quantized)
• Size: 70.51 GB across 15 shards (vs ~470 GB BF16 — a 6.7× reduction)

Why hybrid (tq3a-tq2e)?

The experts dominate the parameter count, so dropping them to 2-bit is where almost all the memory savings come from. Qwen3-235B routes top-8 of 128 experts per token, and that redundancy averages out 2-bit quantization noise cleanly — the same effect that lets gpt-oss-120b (128 experts) hold up at 2-bit while gpt-oss-20b (32 experts) collapses into word-salad. The shared attention path is hit on every token with no expert averaging to hide errors, so it's kept at the safer 3-bit. Routers stay full precision because a wrong expert selection is unrecoverable.

Note: this is an instruct model. Use the chat template (the streaming generator applies it automatically).

Will it fit my Mac?

Don't guess, and don't download 75.7 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      75.7 GB
    experts            71.0 GB  (streamable)
    resident           4.7 GB
  KV cache             188.0 KB/token  (all layers full-attention)

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Quality

Validated with a 6-probe stress suite on a 64 GB Apple Silicon Mac (40 GB expert cache, greedy decode). Five of six probes pass cleanly:

Multi-step arithmetic — usually the soft spot for 2-bit-expert MoEs — was exactly right here, confirming that 2-bit experts hold at Qwen3's 128-expert / top-8 routing (this is the first qwen3_moe model validated under TurboQuant expert streaming). The one blemish was fine-grained factual recall: a specific alphanumeric token from the context came back with one digit changed.

Caveat — exact recall. The observed soft spot is precise factual recall (exact codes, IDs, digit strings) rather than reasoning: in stress testing one alphanumeric needle returned with a single flipped digit (91→51), while reasoning, math, code, and formatting were all correct. Probing showed the 2-bit experts inject a faint prior that only overrides the weakest copy signal — the leading digit of a multi-digit literal; trailing digits and alphabetic spans are recalled perfectly.

Need exact recall? Use the full-3-bit sibling — Qwen3-235B-A22B-Instruct-2507-tq3-g32 (3-bit experts, ~103 GB). Raising the experts to 3-bit removes that prior entirely: it recalls the same needle correctly (RAVEN-stone-91). Choose this hybrid for the smallest footprint (70.5 GB); choose the tq3 build when exact literal recall matters.

Running on limited RAM (expert streaming)

At 70.51 GB the model only fits fully resident on a 96 GB+ Apple Silicon machine. On a 64 GB or 16 GB Mac it still runs by streaming MoE experts from disk: each token pages in only its router-selected experts (LRU-cached), so the 282 big expert tensors are never all in memory at once. Resident memory stays a few GB and output is bit-identical to the fully-resident model.

Measured (greedy decode):

On the 64 GB Mac a 40 GB cache holds ~60% of the ~67 GB of experts, but temporal locality lifts the hit-rate to 94.1% — so once the working set is warm, decode runs at the model's compute-bound ~4–6 tok/s (the math, format, and repetition probes ran at this rate). Throughput is bursty: the first generation and tasks that route into a colder slice of experts still stall on the SSD (the 6-probe run read 269.5 GB total), pulling those tests below 1 tok/s until their experts page in. Peak memory was 46 GB, comfortably inside 64 GB.

On 16 GB it is far more disk-bound: the per-token working set is ~4.1 GB (top-8 × 94 layers × ~5.5 MB/expert), and a 6 GB cache holds less than a fifth of the ~67 GB of experts, so most tokens fall through to the SSD (~724 MB/s here). It runs, but slowly.

Levers that raise throughput:

• More cache — --cache-budget-gb 40-50 on a 64 GB Mac keeps most hot experts resident (the 94% hit-rate above). Bump sudo sysctl iogpu.wired_limit_mb=57344 to go past the ~48 GB default Metal wired cap.
• Faster storage — a Thunderbolt NVMe (~2-3 GB/s) vs the internal SSD directly multiplies streaming speed, since SSD bandwidth is the limiter on cache misses.
• 96 GB+ Mac — skip streaming entirely; load fully resident.

Expert streaming for qwen3_moe ships in TurboQuant-MLX 0.6.0+:

pip install "turboquant-mlx-full>=0.6.0"

python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512 --cache-budget-gb 40

Requirements

# macOS with Apple Silicon (M1/M2/M3/M4)
pip install turboquant-mlx-full mlx-lm

Quick Start

Fully resident (96 GB+ machine), via mlx-lm:

from mlx_lm import load, generate

model, tokenizer = load("manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32")
response = generate(
    model, tokenizer,
    prompt="Write a Python function that returns the nth Fibonacci number.",
    max_tokens=512,
)
print(response)

On 16-64 GB, use the streaming generator shown above.

How It Works

TurboQuant applies:

1. Hadamard rotation — random ±1 scaling to decorrelate weights before quantization
2. Lloyd-Max codebook — optimal scalar quantization via k-means
3. Group-wise scaling — per-group float16 scales for precision

This achieves better quality than standard affine quantization at the same bit-width. MoE models with many experts (here, 128 with top-8 routing) carry enough redundancy to absorb 2-bit expert quantization, while the always-on attention path is kept at 3-bit for safety — the hybrid that this checkpoint ships.

Reproducing the conversion

python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
    --mlx-path /Volumes/SSD/qwen3-235b-tq3a-tq2e-g32 \
    --bits 3 --mlp-bits 2 -g 32 --streaming

--mlp-bits 2 drops the experts to 2-bit while --bits 3 keeps attention at 3-bit; --streaming writes each quantized layer to a shard and frees it, so the full 235B model converts in ~8-12 GB of RAM (peak) — it was produced on a 16 GB Mac mini in ~18 minutes. Point --mlx-path at a drive with ≥70 GB free.

License

Apache 2.0 (same as the base model). Quantization tooling: TurboQuant-MLX.

Copyright 2026 Manjunath Janardhan.

Citation

@article{zandieh2025turboquant,
  title={TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate},
  author={Zandieh, Amir and Daliri, Majid and Hadian, Majid and Mirrokni, Vahab},
  year={2025},
  eprint={2504.19874},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2504.19874}
}