Niwaki-Qwen3.6-30B-A3B-v3.0-gguf

GGUF build of Niwaki-Qwen3.6-30B-A3B-v3.0-mlx — Qwen3.6-35B-A3B with every routed expert stored at its own precision or removed, its program-state tracking intact and ten major languages kept — for llama.cpp. Requires the neopolita-llama.cpp fork at tag v3.0 or later (see Usage): stock llama.cpp cannot represent per-expert precision yet.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.

Project page, with the releases and their measurements side by side: niwakiai.com

File

The same model as the MLX repo. Scored on identical tokens (WikiText-2, 40 windows of 512 tokens), this file reads 6.410 against the MLX artifact's 6.383 (+0.4%), and within 0.5% of it on FLORES-200 in each of English, Spanish, French, German, Russian, Chinese, Japanese, Korean, Arabic and Hindi — the precision layout carries over expert by expert (the routed experts stored at 2-bit sit in Q2_K, those at 3 and 4 bits in Q4_K, quantized with an importance matrix collected on text in those ten languages; the backbone in Q8_0). The program-state battery, run through llama-server on the suite's items, matches the MLX artifact within two items per family: two-hop tracking 1.000 (MLX 0.992), the distractor family 0.983 (0.983), two-step arithmetic 0.550 (0.533), multi-hop recall 0.938 (0.938).

Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 on WikiText-2 under the identical protocol (llama-perplexity, WikiText-2 test, 512-token windows). These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the canonical measurements (task suites, gsm8k, the program-state battery) live there.

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated, one session): 936 tok/s prompt, 64 tok/s decode; the v2.3-4bit UD-Q3K build 1,494 / 82 tok/s in the same session. Each layer runs its experts in two precision tiers, which costs speed against a single-bank file; the file still runs at about 1.5× the decode speed of the intact model's dynamic quants of similar size (UD-IQ3_XXS / UD-IQ3_S: 41–43 tok/s, 785–796 prompt, measured in an earlier session in which the v2.3 build read the same 82 tok/s). The MLX repo decodes at 82 tok/s on the same machine.

Against the earlier Niwaki builds and the intact model

Perplexity under the protocol above (llama-perplexity, 512-token windows, WikiText-2 test and a 512-chunk C4 slice), all measured on the same machine with the same build family:

At the size of v2.3's Q4_K_M build this file reads 13% lower WikiText-2 and 12% lower C4 perplexity. Against the intact model's dynamic quants of similar size it is 2% lower on WikiText-2 and 2% higher on C4 — even on perplexity — while keeping the MLX artifact's program-state behaviour shown above.

Model dimensions

Usage

git clone --branch v3.0 https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion
hf download neopolita/Niwaki-Qwen3.6-30B-A3B-v3.0-gguf Niwaki-Qwen3.6-30B-A3B-v3.0.gguf --local-dir .
./build/bin/llama-cli -m Niwaki-Qwen3.6-30B-A3B-v3.0.gguf
# or serve:
./build/bin/llama-server -m Niwaki-Qwen3.6-30B-A3B-v3.0.gguf

Why a fork

Every routed expert of this model has its own precision, and some are removed. Stock llama.cpp stores a layer's experts in one tensor with one quantization type and expects all of them to exist; the fork lets a qwen35moe layer store its experts in precision tiers — one tensor per quantization type, with a small per-tier table of which experts it holds — under the full router, so the routing is the base model's and a removed expert contributes nothing. Files without tiers, including every earlier Niwaki GGUF, load and run exactly as before. Files produced by the fork's converter load only in the fork; upstreaming is planned.

Canonical benchmarks, the progress table since the first generation, and the MLX-native artifact: Niwaki-Qwen3.6-30B-A3B-v3.0-mlx. Family: v3.0 27B / 24B · v2.3 4-bit / 2-bit · v2.2 4-bit / 2-bit · v2.1 4-bit / 2-bit · v2 4-bit / 2-bit · first-generation 27B-2bit · 19B-2bit · 11B-4bit.

Base model by the Qwen team (Apache 2.0); compression and GGUF builds by the Niwaki project, 2026-09.