Qwen3.6-35B-A3B-tq3a-tqTe-g64

Ternary (1.58-bit) expert TurboQuant quantization of Qwen/Qwen3.6-35B-A3B — 3-bit attention, ternary {-c, 0, +c} experts packed as base-3 trits (group size 64) — produced with TurboQuant-MLX.

A 35B-parameter MoE compressed from ~70 GB (BF16) to 9.4 GB — small enough to run fully resident on a 16 GB Mac mini, with no expert streaming from disk at all.

Documentation: manjunathshiva.github.io/turboquant-mlx: the Qwen3.6-35B-A3B page, getting started, choosing a model for your Mac's memory, guides for serving an OpenAI-compatible API, local coding agents, models bigger than RAM and memory and macOS settings.

Model Details

• Base model: Qwen/Qwen3.6-35B-A3B — Mixture-of-Experts (qwen3_5_moe)
• Architecture: 40 layers (hybrid: 30 GatedDeltaNet linear-attention + 10 full-attention), hidden size 2048, 256 routed experts (top-8) + 1 shared expert, MoE intermediate 512, ~35B total / ~3B active params
• Quantization: TurboQuant (Hadamard rotation + Lloyd-Max codebook), group size 64
  • Attention / dense → 3-bit (q/k/v/o_proj, GDN projections, lm_head)
  • Experts → ternary 1.58-bit (gate/up/down_proj of every routed expert): the data-free {-c, 0, +c} codebook (optimal Lloyd-Max for N(0,1), c ≈ 1.224), with the {0, 1, 2} indices packed as genuine base-3 trits — 20 per uint32 (3²⁰ < 2³²) = ~1.6 bpw, vs 2.0 for a bit-packed 2-bit slot
  • Routers and shared-expert gates → full precision (auto-skipped — never quantized)
• Size: 9.40 GB (vs ~70 GB BF16 — a 7.4× reduction; vs 16.4 GB for the 3-bit-expert sibling)

Why ternary (tq3a-tqTe)?

The 256 routed experts dominate the parameter count, so the memory story is set by how few bits they take. Adding the zero level to a two-sided codebook ({-c, 0, +c} vs {-c, +c}) is what makes sub-2-bit work data-free: a plain 1-bit expert tier collapses into word salad, while ternary lands at ~2-bit reconstruction quality. Packing those three levels in base 3 is a real 1.6 bits/weight, which is what pulls this build under 10 GB and inside a 16 GB machine's resident budget.

Ternary needs expert redundancy to average out its coarser quantization, and this model is the best case in the whole TurboQuant catalog: top-8 of 256 experts per token — twice the redundancy of the 128-expert Qwen3-235B where the same recipe already passes a full stress suite (and 8× the 32-expert models where sub-2-bit demonstrably fails). The shared attention/GDN path is hit on every token with no expert averaging to hide errors, so it stays at the safer 3-bit. Routers stay full precision because a wrong expert selection is unrecoverable.

Note: this is a thinking-capable instruct model. Use the chat template (the generator applies it automatically).

Will it fit my Mac?

Don't guess, and don't download 9.4 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-g64

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      9.4 GB
    experts            7.5 GB  (streamable)
    resident           1.9 GB
  KV cache             20.0 KB/token  (hybrid: 10/40 full-attention)

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-g64 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Running it — fully resident on a 16 GB Mac mini

Requires turboquant-mlx-full ≥ 0.12.3 (base-3 trit decode kernels + the thinking-mode sampling defaults this model card assumes).

pip install "turboquant-mlx-full>=0.30.0" mlx-lm

python -m turboquant_mlx.generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-g64 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512

No --cache-budget-gb, no expert streaming, no external SSD, no sudo — the whole model sits in unified memory under the mini's default Metal wired-memory cap (~10.5 GB). Context growth barely moves the needle: 30 of the 40 layers are GatedDeltaNet with constant-size recurrent state, so only 10 full-attention layers grow KV — measured peak was identical (10.42 GB) at 512 and at 1800 generated tokens. For very long contexts (10K+ tokens) or long-lived server sessions, raise the cap once per boot for headroom:

sudo sysctl -w iogpu.wired_limit_mb=14336   # 14 GiB

Fully resident:

* Estimated for turboquant-mlx-full 0.30.0, not measured. 0.30.0 made ternary experts decode faster: the kernels no longer do an integer division per weight. The estimate comes from kernel timings at this model's expert shapes, calibrated against measured speeds; on NVIDIA Nemotron 3.5 Super VL it predicted 20.4 tok/s, and the build measured 21.0.

Thinking mode on a 1.58-bit build — pick your mode

This is a thinking-capable model, and at 1.58 bits the think phase is the fragile part: it deliberates verbosely and can loop. As of turboquant-mlx-full 0.12.3 the right defaults are built in — pick per task:

• Quick answers, strict formats, math → --no-think. Instant (seconds instead of minutes) and immune to think-loops; also gave the correct answer on math probes where thinking mode wandered.
• Long-form thinking sessions → just run it. This repo ships repetition_penalty: 1.05 in generation_config.json, which the CLI (≥ 0.12.3) applies automatically — in testing it's the one sampling setting that reliably exits <think>. Force off with --rep-penalty 0.
• The CLI also guards against a low-bit quirk where a second </think> re-emits the whole answer (disable with --multi-think if you ever want multiple think blocks).

For an OpenAI-compatible local server (chat UIs, RAG backends — but read "Not for agent harnesses" before wiring it into coding agents):

pip install "turboquant-mlx-full>=0.30.0"
turboquant-serve --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-g64 --kv-bits 8

The server applies the same repetition_penalty default for clients that don't send one (--rep-penalty 0 disables). To serve with thinking off, add --chat-template-args '{"enable_thinking": false}' (or send "chat_template_kwargs": {"enable_thinking": false} per request).

Quality

Validated with a 6-probe stress suite on a 64 GB Apple Silicon Mac (fully resident, temp 0.7). All six probes pass:

Format-test note: in thinking mode this model can spend a long time deliberating on strict-format tasks; with a tight max_tokens cap the generation can end while still inside <think>. That's a thinking-budget effect, not a quantization one — with thinking disabled it answers the same probe perfectly in a fraction of a second. For strict-format or latency-sensitive use, pass --no-think (or give thinking runs ≥4K tokens); see "Thinking mode on a 1.58-bit build" below.

Measured accuracy

Benchmarked on this exact 9.4 GB build (fully resident, Apple Silicon, sampling from this repo's generation_config.json; stratified random samples, seeded):

Protocol notes, in the spirit of honest small-sample benchmarking:

• MMLU-Redux 2.0 questions, stratified across all 57 subjects, using only questions whose re-annotation confirmed the label (error_type == "ok"). The official 93.3 was measured by Qwen on the full set with full thinking; treat the comparison as indicative, not exact.
• 16 of the 171 thinking generations were still inside <think> at the 8192-token budget (scored as-is, mostly wrong). Excluding those budget-capped questions gives 83.6% (127/152) — the cap mainly penalizes hard math subjects where deliberation runs long.
• Net: the 1.58-bit-expert build retains roughly 86–90% of the original's MMLU-Redux score at ~13% of its size. On the paired question subset, thinking adds ~4 points over --no-think (79.8% vs 76.2%).
• GSM8K quantifies the documented math weak spot: two-thirds of grade-school word problems solved in --no-think mode; misses are typically late-stage arithmetic slips, not reasoning failures. Use a less-compressed build when exact arithmetic matters.

Not for agent harnesses (measured)

This build is for chat and one-shot generation. It is not suitable for agent harnesses (Opencode, Aider, Claude Code-style tool loops) — tested directly, not assumed:

• Tool-call syntax works: it emits well-formed OpenAI tool_calls JSON on the first try, and the serving stack is agent-fast (~750 tok/s prefill on an M4 Max, ~3 s/agent-turn, 96–99.9% prompt-cache reuse per turn).
• Tool-use judgment does not survive 1.58-bit: in a fix-the-failing-test task under Opencode it failed 4/4 runs across sampling settings — repeating an identical failed command hundreds of times, or inventing tool arguments (paths, URLs) from unrelated context — and never completed the observe → diagnose → edit loop. Routing more experts at inference (top-12/top-16 instead of top-8) improved literal grounding but not planning, so this is an information floor of the ternary experts, not a sampling or harness issue.
• Control: the 3-bit sibling (tq3-g32) on the identical server + harness + prompt completed the whole task (run tests → find bug → fix → re-run green) in 33 seconds. Multi-step tool use survives 3-bit; it does not survive 1.58-bit.

Practical guidance: use this build for chat, drafting, summarization, Q&A, and one-shot code generation; use the 3-bit sibling (or larger) when a tool loop drives the model. If you do point an agent at it anyway, sandbox the shell — an unreliable model with auto-approved tools will execute hallucinated commands.

Choosing a build

How it was made

python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.6-35B-A3B \
    --mlx-path Qwen3.6-35B-A3B-tq3a-tqTe-g64 \
    --bits 3 --group-size 64 --ternary-experts

Data-free (no calibration set): randomized Hadamard rotation → per-group RMS scaling → Lloyd-Max codebook (ternary for experts) → base-3 trit packing. The 3-entry codebook is self-describing on disk — loaders and Metal kernels switch to base-3 decode automatically.

Links

• Documentation: manjunathshiva.github.io/turboquant-mlx
• TurboQuant-MLX on GitHub (Apache-2.0)
• PyPI: turboquant-mlx-full
• Sibling ternary build: Qwen3-235B-A22B ternary (53 GB, 64 GB Macs)