Niwaki-Qwen3.6-30B-A3B-v3.0-mlx

Qwen3.6-35B-A3B with every routed expert stored at its own precision — 4, 3 or 2 bits — or removed: 10.1 GB of expert storage, 3.4× less than the 8-bit reference, perplexity within 3–8% of it, a task average at its level, the model's program-state tracking intact, and ten major languages kept within 4–21% of the reference. Loads with the shim shipped in this repo; everything else is stock mlx_lm.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts.

A paper with the full method and measurements is coming soon.

Project page, with the releases and their measurements side by side: niwakiai.com

What changed from v2.3

The v2 line removed three quarters of the experts in layers 10–29 and stored every remaining expert at one precision. v3.0 changes the layout: every layer keeps its full router, and each of the 10,240 routed experts is stored at its own precision — 520 at 4-bit, 2,625 at 3-bit, 5,628 at 2-bit — or removed (1,467; a removed expert's routing slot is dropped). At 11% fewer expert bytes than v2.3-4bit, perplexity falls by 12.2% on WikiText-2 and 11.3% on C4, the task average rises from 0.772 to 0.794 (the reference: 0.796), the tracking and distractor families stay at the reference's level on both evaluation seeds, and of ten major languages eight read closer to the reference than v2.3-4bit does, Arabic and Hindi within 0.04 of it (see Languages). The one family where v2.3 stays ahead is two-step arithmetic in generation: 0.97–0.98 there, 0.53–0.62 here, below the reference's own 0.63–0.68.

Benchmarks

Full evaluation protocol: WikiText-2 (145 × 2048-token windows) and C4 (256 × 2048-token windows) perplexity; complete lm-evaluation-harness suites for arc_easy, hellaswag, piqa, winogrande and boolq; gsm8k on 100 problems under lm-eval's default 5-shot prompt (strict match).

The Niwaki line so far

One row per generation at its flagship byte point, all under the protocol above (the first-generation models were re-measured for this table; their own cards carry older 500-sample screens).

The line reads left to right as a trade. v1's flagship spent the fewest bytes and held its benchmarks, but lost the model's program-state tracking; v2 cut deeper and lost gsm8k as well; v2.1 bought both back for 1.5× v1's bytes; v2.2 and v2.3 kept that layout and took 7–8% off perplexity through better kept experts and better recovery training. v3.0 changes the layout itself and, at 11% fewer bytes than v2.3, comes within 3–8% of the reference's perplexity with its task average and tracking at the reference's level and ten major languages within 4–21% of it.

Against keeping every expert at 2-bit

The alternative to removing experts is to keep them all and quantize harder. The fairest such control we can build is the 8-bit reference with only its routed experts recast to 2-bit (group 64) and the backbone untouched: 10.1 GB of expert storage (0.294×), exactly this model's bytes, measured under the identical protocol — once as cast, and once after recovery training. The last three columns are our program-state battery (generation accuracy, range over two evaluation seeds): the value a variable holds through two-hop aliasing, the same past a distractor line, and two-step arithmetic.

At equal bytes this model is ahead of keeping every expert at 2-bit on most measures here, trained or not: perplexity 6–12% lower, a higher task average and masked gsm8k, and tracking at the reference's level where 2-bit noise on every expert breaks it (0.78–0.82). Two-step arithmetic is above the as-cast control and level with the trained one, and the as-cast control's default-prompt gsm8k (0.93) is higher. Against v2.3 it is a split: v2.3's layout carries two-step arithmetic in generation at 0.97–0.98, above the reference, while this model sits below the reference's level — pick v2.3 for exact multi-step arithmetic in generation, this model for everything else.

Languages

FLORES-200 (devtest; each language's 1,012 sentences in 1,024-token windows, identical tokens for every row), perplexity relative to the 8-bit reference — 1.00 means no loss:

These ten languages are the ones this model is built to keep; other languages are not protected the same way. Of ten further languages we measured, Vietnamese, Thai, Belarusian, Malayalam and Punjabi read 1.19–1.31, Greek, Azerbaijani, Icelandic, Hebrew and Turkish 1.43–1.59.

Evaluation methodology

Task average: the complete test sets of the five benchmarks scored by per-choice log-likelihood — arc_easy, hellaswag, and piqa report normalized accuracy (acc_norm); winogrande and boolq report accuracy (acc); the average is the unweighted mean of the five. Perplexity rows use the full window protocol above, identical windows for every row, loss in fp32, no chat template. gsm8k is lm-evaluation-harness's default 5-shot prompt with greedy decoding and strict answer matching, 100 problems; the 30B row caps generation at 512 tokens (in raw few-shot completion it can run past its answer, and mlx_lm's harness adapter applies the stop strings only after generating), the 27B and 24B rows stop on their own; "masked" reruns the same problems with the end-of-sequence tokens masked at sampling (384-token cap). Generation diversity: an 8-prompt battery (code, reasoning, chat, creative; 600-token sampled generations) scored by bigram diversity (distinct bigrams / total bigrams; d2 is a diversity statistic, not a quality score — the reading is closeness to the reference's 0.63 / 0.51), every row measured with the same script, prompts and seeds, the reference alongside.

Model dimensions

Usage (MLX, Apple Silicon)

This checkpoint stores one quantized expert bank per precision in each layer, which stock mlx-lm cannot express — load through the shim in this repo (it builds those banks from config.json, then defers to mlx_lm.load):

# curl -O https://huggingface.co/neopolita/Niwaki-Qwen3.6-30B-A3B-v3.0-mlx/resolve/main/niwaki_v2_load.py
from niwaki_v2_load import load
from mlx_lm import generate

model, tokenizer = load("neopolita/Niwaki-Qwen3.6-30B-A3B-v3.0-mlx")
print(generate(model, tokenizer, prompt="...", max_tokens=256))

GGUF build for llama.cpp, through the neopolita-llama.cpp fork at tag v3.0 or later (it reads this model's per-expert precision): Niwaki-Qwen3.6-30B-A3B-v3.0-gguf — one 14.1 GB file that reads within 0.5% of this repo's perplexity on identical tokens, in English and in each of the ten languages above.

Notes and limitations

• Compression trades quality: perplexity sits 3–8% above the reference's, and two-step arithmetic in generation falls below the reference's level (0.53–0.62 against 0.63–0.68), where v2.3 sits above it. Choose the family member that fits your memory budget.
• Built to keep English, Spanish, French, German, Russian, Chinese, Japanese, Korean, Arabic and Hindi (see Languages); other languages may lose more, and the benchmarks above are English. The base model's biases are inherited and rare-domain behaviour is less tested.
• Speed: MLX decode on an M4 Max runs at 82 tok/s against the 8-bit reference's 89 tok/s in the same session (256 tokens after an 80-token prompt, median of 5 runs; ranges 79–83 and 80–93) — about 7% below the reference's decode speed at 35% of its peak memory (12.1 vs 34.5 GiB).
• The chat template works as on the base model (thinking on or off). In raw completion mode the model continues few-shot patterns like a base model does; evaluate with stop sequences.
• Part of the Niwaki family: the v3.0 siblings Niwaki-Qwen3.6-27B-A3B-v3.0-mlx / Niwaki-Qwen3.6-24B-A3B-v3.0-mlx, the v2.3 models 4-bit / 2-bit, the v2.2 models 4-bit / 2-bit, the v2.1 models 4-bit / 2-bit, the v2 models 4-bit / 2-bit, and the first-generation 27B / 19B / 11B models.

Base model by the Qwen team (Apache 2.0); 8-bit MLX conversion by mlx-community; compression by the Niwaki project, 2026-09.