leonsarmiento/Qwen3.6-35B-A3B-3bit-XL-mlx
This model was converted to MLX format from Qwen/Qwen3.6-35B-A3B using BaseQuant_XL 3-bit mixed quantization — the compact XL variant built for RAM-constrained Macs. The vision encoder is preserved (quantized at 6-bit for quality), making this a full multimodal model.
BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router gate, shared expert gate, shared expert, and lm_head — while applying aggressive quantization to the bulk parameters. This is the same layer assignment as the 6-bit XL variant — only the routed experts (switch_mlp) drop from 6-bit to 3-bit.
Qwen3.6-35B-A3B is a 35B-parameter multimodal MoE (Mixture of Experts) model with 256 experts (8 active per token + 1 shared expert), hybrid full + linear (Gated DeltaNet) attention, and a vision encoder. Despite 35B total parameters, only ~3B are activated per token for efficient inference.
Use with mlx
pip install -U mlx-vlm
python -m mlx_vlm.generate --model leonsarmiento/Qwen3.6-35B-A3B-3bit-XL-mlx --max-tokens 256 --temperature 0.6 --top-p 0.95 --prompt "Hello"
BaseQuant_XL Quantization Strategy
| Bit Depth | Layers | Rationale |
|---|---|---|
| bf16 (unquantized) | mlp.gate (router), shared_expert_gate, lm_head, shared_expert |
Routing decisions and shared computation path — errors here are qualitatively different from precision loss |
| 8-bit | embed_tokens, self_attn (full attention), linear_attn (DeltaNet) |
Every-token layers with moderate sensitivity — 8-bit is near-lossless |
| 6-bit | vision_tower |
Vision quality kept at the proven 6-bit level; negligible share of total parameters |
| 3-bit | switch_mlp (routed experts) |
The bulk of parameters, only 8 of 256 experts active per token — where the size win comes from |
Quantization Details
| Layer | Bits | Group Size |
|---|---|---|
mlp.gate (router) |
bf16 | — |
shared_expert_gate |
bf16 | — |
lm_head |
bf16 | — |
shared_expert |
bf16 | — |
embed_tokens |
8 | 64 |
self_attn (full attention) |
8 | 64 |
linear_attn (DeltaNet) |
8 | 64 |
vision_tower |
6 | 64 |
switch_mlp (routed experts) |
3 | 64 |
| Default fallback | 8 | 64 |
- Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
- Bits per weight: 4.055
- Total size: ~17 GB (4 shards)
- Group size: 64
- Method: Custom
quant_predicateviamlx_vlm
Recommended Inference Parameters
Thinking Mode (default)
| Parameter | Value |
|---|---|
temperature |
1.0 |
top_p |
0.95 |
top_k |
20 |
presence_penalty |
1.5 |
reasoning_parser |
qwen3 |
Instruct Mode (non-thinking)
| Parameter | Value |
|---|---|
temperature |
0.6 |
top_p |
0.95 |
top_k |
20 |
presence_penalty |
1.5 |
enable_thinking |
false |
Instruct Benchmarks
| Model | MMLU | MMLU_PRO | HUMANEVAL | MBPP | Avg |
|---|---|---|---|---|---|
| Huihui-Gemma-4 26B 3bit-XL | 78.0% | 71.0% | 98.0% | 82.0% | 82.3% |
| Gemma-4 26B 3bit-XL | 80.0% | 65.0% | 98.0% | 86.0% | 82.3% |
| Qwen3.8-27B 3bit-mtp | 80.0% | 61.0% | 94.0% | 70.0% | 76.3% |
| Qwen3.6-35B 3bit-XL | 82.0% | 63.0% | 66.0% | 80.0% | 72.8% |
| Huihui-Qwen3.6-35B 3bit-XL | 83.0% | 63.0% | 62.0% | 76.0% | 71.0% |
Instruct mode (thinking off). Sampled: MMLU 100/14042, MMLU_PRO 100/12032, HUMANEVAL 50/164, MBPP 50/500. Wall times not directly comparable across architectures (dense vs MoE active params).
Highlights: MMLU 82% — knowledge recall holds at 3-bit. Code is the 3-bit trade-off: HUMANEVAL 66% (vs 74% for the 6-bit XL sibling).