Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-gguf

GGUF builds of Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-mlx — Qwen3.6-35B-A3B pruned to 22.6B total / ~3.8B active parameters, with the model's program-state tracking and few-shot reasoning intact — for llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the neopolita-llama.cpp fork (see Usage): stock llama.cpp cannot represent this model's layout yet.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.

Project page, with the releases and their measurements side by side: niwakiai.com

Files

These files are the GGUF counterpart of the 2-bit MLX artifact: its experts re-quantized onto llama.cpp's 3- and 4-bit carriers, which makes them the same size as the sibling Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-gguf builds (8.39 / 8.31 under the identical protocol). At these sizes the sibling is the better choice; pick this repo when you want the 2-bit artifact's exact function in llama.cpp.

v2.3 keeps v2.2's layout and sizes exactly; what changed is only the recovery training that follows the pruning. The earlier generations' 2-bit GGUF builds under the identical protocol:

This build beats the v2.2-2bit UD-Q3K build (9.69) by 2.3% at the same size (5% on the C4 slice) and the v2.1 build by 9%.

Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 under the identical protocol (llama-perplexity, WikiText-2 test, 512-token windows). These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the relative standings match across both, and the canonical measurements (task suites, gsm8k, the program-state battery) live there.

Generation battery (measured on the canonical MLX weights; reference scores 0.63 / 0.51 under the identical battery): bigram-diversity avg/min = 0.60 / 0.44 across an 8-prompt code/reasoning/chat/creative battery.

The recommended UD-Q3K build is quantized structure-aware (importance matrices calibrated on a mixed web/code/chat/reasoning corpus), mirroring the artifact's native allocation: the always-active backbone (attention, shared experts, embeddings) is kept at high precision (Q6_K), the untouched layers' routed experts ride a compact carrier (q3_k, imatrix-guided), and the 64-expert banks of layers 10–29 — the part the MLX artifact never takes below 4-bit either — stay at q4_k. It matches uniform Q4_K_M quality (9.53) at ~20% fewer bytes.

Format note: no zero-valued pad: layers 10–29 store exactly their 64 kept experts (the fork reads a per-layer expert count), so the file carries only experts that exist.

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated; measured on the v2.2 builds, whose file layout and sizes these files share exactly): UD-Q3K 1,445 tok/s prompt, 82 tok/s decode; Q4_K_M 1,465 / 91 tok/s — the Q4_K kernels are faster than Q3_K on Metal.

Against the intact model at the same size

The alternative to pruning is to quantize the intact 35B model harder. Unsloth's dynamic quants of Qwen3.6-35B-A3B that bracket these files, measured on the same machine, build and protocol (llama-perplexity, 512-token windows, WikiText-2 test and a 512-chunk C4 slice; llama-bench 512-token prompt / 128 generated, Metal):

At equal size the intact model's dynamic quants read 19–23% lower WikiText-2 and 18–22% lower C4 perplexity than these builds (the 4-bit sibling's builds, at the same sizes, narrow that to 8–13%); these builds run 1.7–2.1× faster on Metal, on prompt processing and decode alike (the Q3_K / Q4_K kernels against the IQ kernels, and fewer routed-expert bytes per token). Pick by what you are short of: quality at equal memory favours the intact quants, throughput at equal memory favours these files. The function-level comparison (task suites, gsm8k, program-state battery) is on the MLX card.

Model dimensions

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-gguf Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-UD-Q3K.gguf

Why a fork

Layers 10–29 of this model keep 64 of their 256 routed experts and route only over them. Stock llama.cpp requires every MoE layer of a model to carry the same number of experts; the fork lets a qwen35moe layer's expert count follow its tensors, with a router of one row per kept expert (its softmax is exactly the renormalised routing of the MLX artifact). Files produced by the fork's converter load only in the fork; upstreaming is planned.

Canonical benchmarks, the progress table since the first generation, and the MLX-native artifact: Qwen3.6-23B-A3B-Niwaki-v2.3-2bit-mlx. Family: v2.3-4bit · v2.2 4-bit / 2-bit · v2.1 4-bit / 2-bit · v2 4-bit / 2-bit · first-generation 27B-2bit · 19B-2bit · 11B-4bit.

Base model by the Qwen team (Apache 2.0); pruning, recovery training, and GGUF builds by the Niwaki project, 2026-09.