8-bit base mixed-precision quantization of Qwen/Qwen3.6-35B-A3B for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.
| Original (BF16) | This Model | |
|---|---|---|
| Size | ~66 GB | 36 GB |
| Format | SafeTensors (sharded) | SafeTensors (sharded) |
| Precision | BF16 uniform | Mixed 8-bit + BF16 |
| Repo | GGUF Equivalent | Size | Decode (tok/s) |
|---|---|---|---|
| Brooooooklyn/Qwen3.6-35B-A3B-UD-Q2_K_XL-mlx | UD-Q2_K_XL | 14 GB | 63.7 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-Q3_K_XL-mlx | UD-Q3_K_XL | 18 GB | 59.2 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-MXFP4_K_XL-mlx | — | 21 GB | 54.4 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-NVFP4_K_XL-mlx | — | 22 GB | 59.1 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-Q4_K_XL-mlx | UD-Q4_K_XL | 22 GB | 55.1 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-Q5_K_XL-mlx | UD-Q5_K_XL | 26 GB | 51.8 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-Q6_K_XL-mlx | UD-Q6_K_XL | 31 GB | 54.4 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-MXFP8_K_XL-mlx | — | 35 GB | 47.6 |
| Brooooooklyn/Qwen3.6-35B-A3B-UD-Q8_K_XL-mlx (this model) | UD-Q8_K_XL | 36 GB | 45.9 |
Benchmarked on Apple M3 Max 128GB via examples/lm.ts (best decode tok/s across turns 2–4, steady-state).
Steady-state decode: 45.9 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low').
Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), and the compiled C++ forward graph fuses the per-layer dispatch.
| Weight | Bits | Rationale |
|---|---|---|
embed_tokens |
8-bit | KLD ~0.15 — very low sensitivity |
lm_head |
8-bit | KLD ~0.05 — safest tensor |
self_attn.q/k/v_proj |
8-bit + AWQ | KLD ~1.5–2.9, AWQ via layernorm |
linear_attn.in_proj_qkv/z |
8-bit + AWQ | KLD ~2.9, AWQ via layernorm |
self_attn.o_proj |
bf16 | NOT AWQ-correctable |
linear_attn.out_proj |
bf16 | KLD ~6.0 — worst tensor |
down_proj |
8-bit | "Slightly more sensitive" |
gate_proj, up_proj |
8-bit | base bits |
| Router gates | 8-bit | MoE routing accuracy |
| GDN params (A_log, etc) | bf16 | State-space dynamics |
Based on Unsloth Dynamic 2.0 per-tensor KLD analysis. Sensitive layers get higher bits with AWQ correction, while the bulk of FFN expert weights are aggressively quantized. imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead).
AWQ-correctable projections (q/k/v, in_proj_qkv/z) are quantized at 8-bit via input_layernorm. Non-AWQ-correctable projections (o_proj, out_proj) are kept at bf16 — their inputs come from attention/GDN computation, not from a norm layer.
| Parameter | Value |
|---|---|
| Total parameters | 35.9B (3B active per token) |
| Hidden size | 2,048 |
| Layers | 40 (30 linear + 10 full attention) |
| Attention heads | 16 (2 KV heads, GQA 8:1) |
| Head dimension | 256 |
| Experts | 256 per MoE layer, top-8 routing |
| Vocab size | 248,320 |
| Max context | 262,144 tokens |
import { loadSession } from '@mlx-node/lm';
const session = await loadSession('./Qwen3.6-35B-A3B-UD-Q8_K_XL-mlx');
for await (const event of session.sendStream('Explain the hybrid attention mechanism in Qwen3.6.', {
config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
if (!event.done) process.stdout.write(event.text);
}
mlx convert \
-i Qwen3.6-35B-A3B \
-o Qwen3.6-35B-A3B-UD-Q8_K_XL-mlx \
-q --q-bits 8 --q-recipe unsloth \
--imatrix-path imatrix_unsloth.gguf
Apache 2.0 (inherited from base model).
model-00006-of-00008.safetensors 5.4 GBmodel-00003-of-00008.safetensors 5.4 GBmodel-00005-of-00008.safetensors 5.4 GBmodel-00001-of-00008.safetensors 5.3 GBmodel-00004-of-00008.safetensors 5.2 GBmodel-00007-of-00008.safetensors 5.2 GBmodel-00002-of-00008.safetensors 5.1 GBmodel-00008-of-00008.safetensors 1.2 GBcd ~/mlx-node-qwen36-test && MAX_NEW_TOKENS=512 RUNS=3 npm run benchView run on LocalMaxxing
Median of 1 community runs on 1 setups, from LocalMaxxing.
Run it yourself: setup guideQwen/Qwen3.6-35B-A3B