Qwen3.6-35B-A3B-tq3-g32

TurboQuant 3-bit quantized (group size 32) version of Qwen/Qwen3.6-35B-A3B, produced with TurboQuant-MLX.

This is the text-only language model extracted from the original multimodal checkpoint — the vision tower is dropped during conversion.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Qwen3.6-35B-A3B page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, models bigger than RAM and memory and macOS settings.

Model Details

• Base model: Qwen/Qwen3.6-35B-A3B — a hybrid linear-attention + Mixture-of-Experts model (qwen3_5_moe)
• Architecture: 40 layers, hidden size 2048, 256 routed experts (top-8) + 1 shared expert, ~35B total / ~3B active params
• Quantization: TurboQuant 3-bit (Hadamard rotation + Lloyd-Max codebook), group size 32
• Router precision: MoE routers and the shared-expert gate are kept at full precision
• Size: ~16 GB (vs ~72 GB BF16)

Will it fit my Mac?

Don't guess, and don't download 16.9 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      16.9 GB
    experts            15.0 GB  (streamable)
    resident           2.0 GB
  KV cache             20.0 KB/token  (hybrid: 10/40 full-attention)

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Measured performance

Apple M4 Max, 64 GB (model fully resident, fp16 KV cache):

The high decode throughput comes from MoE sparsity — only ~3B of 35B params are active per token. The 256-expert redundancy lets 3-bit quantization preserve quality with no observed coherence loss on code, math, and factual prompts.

Note: this is a thinking-mode model — it emits a reasoning trace before the final answer, so use a generous --max-tokens (e.g. 512+) for tasks that need a concluding answer.

Agent harnesses (Opencode, Aider, Claude Code) — measured pass ✅

This is the 35B build to use for tool-driven / agentic work. Tested head-to-head under Opencode against a local turboquant-serve endpoint (thinking off, temp 0.7): given a repo with a failing pytest suite, this build ran the test command it was given, read the source, made the correct minimal fix, and re-ran the tests to green — end-to-end in 33 seconds. The 1.58-bit sibling failed the identical task 4/4 times (details on its card) — multi-step tool use survives 3-bit, not 1.58-bit.

Serve it for agents like this (requires turboquant-mlx-full[serve] >= 0.12.3):

turboquant-serve --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --kv-bits 8 --temp 0.7 --top-p 0.8 --top-k 20 \
    --chat-template-args '{"enable_thinking": false}'

Agent-use notes:

• Pass sampling defaults on the server (as above). Many agent clients send no temperature, and the server's fallback is greedy decoding — fine for chat, but worth avoiding in long tool loops.
• Fully resident needs ~18 GB of wired memory (32 GB Macs comfortable; 24 GB with the sysctl bump). On a 16 GB machine it runs via expert streaming (below) — for agent use there, prefer native routing (--max-active-experts 0): the K-reduction shortcut is validated byte-identical for chat but not for long tool-calling loops, and expect patience on the first turn (agent harnesses send large system prompts, which stream-prefill slowly).
• If the client's model field is configurable, point it at default_model — the server serves the CLI-loaded model under that id.

Running on 16 GB RAM (expert streaming + K-reduction)

The model is ~16 GB on disk, so it won't fit fully resident in 16 GB of RAM alongside the OS (resident decode peaks at ~18 GB). It still runs on a 16 GB Mac mini by streaming MoE experts from disk: for each token only the router-selected experts are paged in (LRU-cached), so the big 256-expert tensors are never all in memory at once. Resident memory stays a few GB.

Two streaming optimizations (inspired by Flash-MoE and Apple's LLM in a Flash) make this fast:

• K-reduction (--max-active-experts 4, default in 0.10.0+): caps router top-8 → top-4 on every MoE block. The top-4 experts dominate routing so heavily that this produced byte-identical output to native top-8 across our 6-test stress harness (code, math, needle-in-haystack, JSON format, long-essay, repetition) — while halving the experts streamed from disk per token. Raise to 6 for extra margin, or 0 for native routing.
• Auto page-cache: the loader compares model size to RAM and, on a 16 GB box (model ≈ RAM), keeps F_NOCACHE on so the OS page cache doesn't thrash the live working set. No flag needed; override with --use-page-cache / --no-page-cache.

Measured on a base Apple M4 Mac mini, 16 GB, with the model on an external spinning Seagate HDD — worst-case random-read storage; an SSD is faster — over a 256-token generation:

A larger LRU cache keeps more experts resident, but the hit-rate saturates near ~88% at ~8 GB — the set of distinct experts touched in a generation doesn't fill more than ~8 GB, so budgets beyond 8 give no speedup (and eat headroom you'd rather keep for KV on long prompts). --cache-budget-gb 8 --max-active-experts 4 is the sweet spot. Raising to top-6 reads ~20% more for no measurable quality gain. The disk is the limiter — ~10 tok/s here off a spinning HDD; moving the model to an SSD removes the per-expert seek cost.

Throughput rises as the cache warms. The table above is the cold-start (256-token) rate. Over a longer generation the LRU fills with the experts this prompt actually routes to, so the hit-rate climbs and per-token disk traffic drops. On the same --cache-budget-gb 8 --max-active-experts 4 config, a ~2.6K-token generation reached ~13 tok/s end-to-end at a 94% hit-rate (~14.5 MB read/token, vs ~28 MB cold) — so a sustained chat or coding session runs noticeably faster than the cold-start number, with peak RSS still ~9.5 GB. Give thinking-mode tasks a generous --max-tokens (4096+) so the reasoning trace completes and the cache gets time to warm.

Expert streaming with K-reduction ships in TurboQuant-MLX 0.10.0+:

pip install "turboquant-mlx-full>=0.10.0"

python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --prompt "Write a Python function that merges two sorted lists." \
    --max-tokens 512 \
    --cache-budget-gb 8 \
    --max-active-experts 4

You can also serve the streamed model to an OpenAI-compatible endpoint (Claude Code, Aider, etc.) with turboquant-serve (add --cache-budget-gb 8 --max-active-experts 4, requires 0.11.0+).

Requirements

# macOS with Apple Silicon (M1/M2/M3/M4)
pip install turboquant-mlx-full mlx-lm

Quick Start

python -m turboquant_mlx.generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --prompt "Write a Python function that returns the nth Fibonacci number." \
    --max-tokens 512 \
    --temp 0.7

With KV-cache compression (for long context)

python -m turboquant_mlx.generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
    --prompt "Your long prompt here..." \
    --max-tokens 512 \
    --kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128

How It Works

TurboQuant applies:

1. Hadamard rotation — random ±1 scaling to decorrelate weights before quantization
2. Lloyd-Max codebook — optimal scalar quantization via k-means
3. Group-wise scaling — per-group float16 scales for precision

This achieves better quality than standard affine quantization at the same bit-width. MoE models with many experts (here, 256) have enough redundancy to absorb 3-bit quantization noise cleanly.

License

Apache 2.0 (same as the base model). Quantization tooling: TurboQuant-MLX.

Copyright 2026 Manjunath Janardhan.

Citation

@article{zandieh2025turboquant,
  title={TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate},
  author={Zandieh, Amir and Daliri, Majid and Hadian, Majid and Mirrokni, Vahab},
  year={2025},
  eprint={2504.19874},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2504.19874}
}