Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-mlx

Qwen3.6-35B-A3B pruned to 22.6B total / ~3.8B active parameters, stored at 2-bit (the 64 kept experts of layers 10–29 at 4-bit) — 4.7× less expert memory than the 8-bit reference — with the model's program-state tracking and few-shot reasoning intact, and lower perplexity than v2.1 at identical bytes. Loads with the shim shipped in this repo; everything else is stock mlx_lm.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts.

A paper with the full method and measurements is coming soon.

What changed from v2.1

Same layout, same byte budget: layers 10–29 keep 64 of their 256 experts and route only over them, the other twenty layers are untouched. v2.2 changes only which 64 experts each of those layers keeps. That choice alone lowers perplexity by 4–7% on both corpora, lifts the task average and holds the program-state battery (variable tracking, two-step arithmetic, multi-hop recall, distractor lines) at the reference's level on both evaluation seeds.

Benchmarks

Full evaluation protocol: WikiText-2 (145 × 2048-token windows) and C4 (256 × 2048-token windows) perplexity; complete lm-evaluation-harness suites for arc_easy, hellaswag, piqa, winogrande and boolq; gsm8k on 100 problems under lm-eval's default 5-shot prompt (strict match).

† gsm8k with the end-of-turn token masked at sampling: this model stops early on 8 of the 100 few-shot prompts under the default protocol (empty responses); with the token masked it answers 87 correctly. The 4-bit sibling does not show this (0.86 / 0.85).

The Niwaki line so far, 2-bit

One row per generation at the 2-bit byte point, all under the protocol above (the first-generation model was re-measured for this table; its own card carries older 500-sample screens).

The line reads as a trade: v1 and v2 spent the fewest bytes and lost the model's program-state tracking — the value a variable holds through two-hop aliasing or past a distractor line — and v1 half of its gsm8k score; v2.1 bought both back for 1.45× the bytes; v2.2 keeps v2.1's bytes and recovers a further 4–7% of perplexity and 1.3 task points from a better choice of kept experts.

Against quantizing the intact model

The alternative to pruning is to keep every expert and quantize harder. The fairest such control we can build is the 8-bit reference with only its routed experts recast to 2-bit (group 64, the same quantizer as this repo) and the backbone untouched: 10.1 GB of expert storage (0.294×), which is the smallest an intact 2-bit cast gets in MLX — this model spends 28% fewer bytes than that. Measured under the identical protocol; the last three columns are our program-state battery (generation accuracy, range over two evaluation seeds): the value a variable holds through two-hop aliasing, the same past a distractor line, and two-step arithmetic.

Read it as a split. Above 0.29× the intact model's 2-bit cast is the stronger artifact on perplexity, the task average and few-shot gsm8k, while the pruned models are the stronger ones on program-state tracking and two-step arithmetic — 2-bit noise on every expert degrades exact state carrying, which the pruned models keep at or above the reference's level. Below 0.29× there is no intact-model equivalent in MLX; this model is the family's answer at that byte point.

Evaluation methodology

Task average: the complete test sets of the five benchmarks scored by per-choice log-likelihood — arc_easy, hellaswag, and piqa report normalized accuracy (acc_norm); winogrande and boolq report accuracy (acc); the average is the unweighted mean of the five. Perplexity rows use the full window protocol above, identical windows for every row, loss in fp32, no chat template. gsm8k is lm-evaluation-harness's default 5-shot prompt with greedy decoding and strict answer matching, 100 problems. Generation diversity: an 8-prompt battery (code, reasoning, chat, creative; 600-token sampled generations) scored by bigram diversity (distinct bigrams / total bigrams; d2 is a diversity statistic, not a quality score — the reading is closeness to the reference's 0.63 / 0.51), every row measured with the same script, prompts and seeds, the reference alongside.

Model dimensions

Usage (MLX, Apple Silicon)

This checkpoint stores compact 64-expert banks for layers 10–29, which stock mlx-lm cannot express — load through the shim in this repo (it rebuilds those banks from config.json, then defers to mlx_lm.load):

# curl -O https://huggingface.co/neopolita/Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-mlx/resolve/main/niwaki_v2_load.py
from niwaki_v2_load import load
from mlx_lm import generate

model, tokenizer = load("neopolita/Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-mlx")
print(generate(model, tokenizer, prompt="...", max_tokens=256))

GGUF builds for llama.cpp (through the neopolita-llama.cpp fork, which reads this model's per-layer expert counts): Qwen3.6-23B-A3B-Niwaki-v2.2-2bit-gguf (UD-Q3K recommended + Q4_K_M).

Notes and limitations

• Compression trades quality: this model retains ~96% of the reference's task average and ~88% of its gsm8k score (masked protocol). Choose the family member that fits your memory budget.
• Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour is less tested.
• No latency claim: all 40 layers execute with the reference's active parameter count. This is a memory artifact.
• The chat template works as on the base model (thinking on or off). In raw completion mode the model continues few-shot patterns like a base model does and may stop at an end-of-turn token early; evaluate with stop sequences or with that token masked.
• Part of the Niwaki family: Qwen3.6-23B-A3B-Niwaki-v2.2-4bit-mlx (sibling), the v2.1 models 4-bit / 2-bit, the v2 models 4-bit / 2-bit, and the first-generation 27B / 19B / 11B models.

Base model by the Qwen team (Apache 2.0); 8-bit MLX conversion by mlx-community; compression and distillation by the Niwaki project, 2026-09.