Qwen3.6-27B-A3B-Niwaki-2bit-mlx
Qwen3.6-35B-A3B pruned to 27B total / 3.6B active
parameters, stored at 2-bit — 4.5× less expert memory than
the 8-bit reference. Stock mlx_lm, no custom code.
Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts: every routed expert is individually width-pruned using statistics of its routed tokens over a mixed web/code/chat/reasoning corpus, reconstructed to compensate, and stored at low precision. The result is an ordinary MoE — smaller, same architecture, standard fused kernels.
A paper with the full method and measurements is coming soon.
Benchmarks
Full evaluation protocol: WikiText-2 perplexity over 256 × 2048-token windows; task average over the complete arc_easy, hellaswag, piqa, winogrande, and boolq test sets, paired against the reference.
| model | expert storage | wt2 ppl ↓ | task avg ↑ | retention | generation d2 (avg/min) |
|---|---|---|---|---|---|
| reference (8-bit) | 34.2 GB | 6.76 | 0.796 | 100% | 0.89 / 0.77 |
| this model | 7.5 GB (0.221×) | 9.89 | 0.769 | 96.7% | 0.89 / 0.79 |
Generation quality is measured, not assumed: an 8-prompt battery (code, reasoning, chat, creative; 600-token sampled generations) scored by bigram diversity.
Model dimensions
| total parameters | 27B (from 35B) |
| active parameters per token | ~3.6B (from ~3.8B: 2.8B backbone + 8 pruned experts) |
| layers / routed experts / top-k | 40 / 256 / 8 |
| expert intermediate size | 384 (from 512) |
| expert precision | 2-bit (group 64) |
| backbone precision | 8-bit (untouched) |
| whole-model size on disk | ~10.5 GB |
| runs comfortably in | 16 GB unified memory |
Usage (MLX, Apple Silicon)
from mlx_lm import load, generate
model, tokenizer = load("neopolita/Qwen3.6-27B-A3B-Niwaki-2bit-mlx")
print(generate(model, tokenizer, prompt="...", max_tokens=256))
GGUF builds for llama.cpp: Qwen3.6-27B-A3B-Niwaki-2bit-GGUF (UD-Q3K recommended + Q4_K_M).
Notes and limitations
- Compression trades quality: this model sits at ~97% of the reference task average and 1.46× its perplexity. Choose the family member that fits your memory budget.
- Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour of pruned experts is less tested.
- Part of the Niwaki family: 19B-A3B-Niwaki-2bit · 11B-A3B-Niwaki-4bit.
Base model by the Qwen team (Apache 2.0); 8-bit MLX conversion by mlx-community; pruning and distillation by the Niwaki project, 2026-08. Method details: paper coming soon.