Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-mlx

Qwen3.6-35B-A3B pruned to 22.6B total / ~3.8B active parameters, stored at 2-bit (the 64 kept experts of layers 10–29 at 4-bit) — 4.7× less expert memory than the 8-bit reference — with the model's program-state tracking and few-shot reasoning intact, and lower perplexity than v2.2 at identical bytes. Loads with the shim shipped in this repo; everything else is stock mlx_lm.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts.

A paper with the full method and measurements is coming soon.

Project page, with the releases and their measurements side by side: niwakiai.com

What changed from v2.2

Same layout, same byte budget, same kept experts: layers 10–29 keep 64 of their 256 experts and route only over them, the other twenty layers are untouched. v2.3 changes only the recovery training that follows the pruning. That alone lowers perplexity by 2.0% on WikiText-2 and 4.9% on C4, holds the task average, lifts multi-hop recall from 0.69 to 0.88 — above the reference's 0.62 — and takes two-step arithmetic in generation from 0.83–0.90 to 0.92–0.93. It also removes v2.2-2bit's one known flaw: that model stopped early on 8 of 100 few-shot gsm8k prompts, this one stops early on none, so its gsm8k score needs no masked protocol.

Benchmarks

Full evaluation protocol: WikiText-2 (145 × 2048-token windows) and C4 (256 × 2048-token windows) perplexity; complete lm-evaluation-harness suites for arc_easy, hellaswag, piqa, winogrande and boolq; gsm8k on 100 problems under lm-eval's default 5-shot prompt (strict match).

† v2.2-2bit's score with the end-of-turn token masked at sampling; that model stopped early on 8 of the 100 few-shot prompts. This model does not (0 early stops), and its 0.85 is the plain default-protocol score.

The Niwaki line so far, 2-bit

One row per generation at the 2-bit byte point, all under the protocol above (the first-generation model was re-measured for this table; its own card carries older 500-sample screens).

The line reads as a trade: v1 and v2 spent the fewest bytes and lost the model's program-state tracking — the value a variable holds through two-hop aliasing or past a distractor line — and v1 half of its gsm8k score; v2.1 bought both back for 1.45× the bytes; v2.2 kept v2.1's bytes and recovered 4–7% of perplexity from a better choice of kept experts; v2.3 keeps v2.2's layout and experts and takes another 2–5% from better recovery training, with multi-hop recall now above the reference's and no early stops.

Against quantizing the intact model

The alternative to pruning is to keep every expert and quantize harder. The fairest such control we can build is the 8-bit reference with only its routed experts recast to 2-bit (group 64, the same quantizer as this repo) and the backbone untouched: 10.1 GB of expert storage (0.294×), which is the smallest an intact 2-bit cast gets in MLX — this model spends 28% fewer bytes than that. Measured under the identical protocol, once as cast and once after the same recovery training this model received; the last three columns are our program-state battery (generation accuracy, range over two evaluation seeds): the value a variable holds through two-hop aliasing, the same past a distractor line, and two-step arithmetic.

Read it as a split. Above 0.29× the intact model with 2-bit experts is the stronger artifact on perplexity and the task average, and after recovery training it matches the reference's task average; what it does not recover, trained or not, is program-state function — two-step arithmetic at 0.37–0.60 and two-hop tracking at 0.78–0.82, against 0.92–0.93 and 0.97–0.98 here — because 2-bit noise on every expert breaks exact state carrying, and this model's 64 kept experts per pruned layer stay at 4-bit. Below 0.29× there is no intact-model equivalent in MLX; this model is the family's answer at that byte point.

Evaluation methodology

Task average: the complete test sets of the five benchmarks scored by per-choice log-likelihood — arc_easy, hellaswag, and piqa report normalized accuracy (acc_norm); winogrande and boolq report accuracy (acc); the average is the unweighted mean of the five. Perplexity rows use the full window protocol above, identical windows for every row, loss in fp32, no chat template. gsm8k is lm-evaluation-harness's default 5-shot prompt with greedy decoding and strict answer matching, 100 problems. Generation diversity: an 8-prompt battery (code, reasoning, chat, creative; 600-token sampled generations) scored by bigram diversity (distinct bigrams / total bigrams; d2 is a diversity statistic, not a quality score — the reading is closeness to the reference's 0.63 / 0.51), every row measured with the same script, prompts and seeds, the reference alongside.

Model dimensions

Usage (MLX, Apple Silicon)

This checkpoint stores compact 64-expert banks for layers 10–29, which stock mlx-lm cannot express — load through the shim in this repo (it rebuilds those banks from config.json, then defers to mlx_lm.load):

# curl -O https://huggingface.co/neopolita/Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-mlx/resolve/main/niwaki_v2_load.py
from niwaki_v2_load import load
from mlx_lm import generate

model, tokenizer = load("neopolita/Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-mlx")
print(generate(model, tokenizer, prompt="...", max_tokens=256))

GGUF builds for llama.cpp (through the neopolita-llama.cpp fork, which reads this model's per-layer expert counts): Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-gguf (UD-Q3K recommended + Q4_K_M).

Notes and limitations

• Compression trades quality: this model retains ~96% of the reference's task average and ~86% of its gsm8k score. Choose the family member that fits your memory budget.
• Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour is less tested.
• No latency claim: all 40 layers execute with the reference's active parameter count. This is a memory artifact.
• The chat template works as on the base model (thinking on or off). In raw completion mode the model continues few-shot patterns like a base model does; evaluate with stop sequences.
• Part of the Niwaki family: Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-mlx (sibling), the v2.2 models 4-bit / 2-bit, the v2.1 models 4-bit / 2-bit, the v2 models 4-bit / 2-bit, and the first-generation 27B / 19B / 11B models.

Base model by the Qwen team (Apache 2.0); 8-bit MLX conversion by mlx-community; compression and distillation by the Niwaki project, 2026-09.