⚠️ Superseded by v2

Qwen3.6-35B-A3B-REAP-48-v2-GGUF is the same model at the same file size, quantized with a measured per-tensor bit allocation instead of this one’s hand-written recipe. Against this file, measured on the same machine and corpus: mean KL-divergence −42.5%, median −52.4%, top-1 agreement with the f16 weights 94.7% → 96.2%, and decode throughput on an M4 Mac Mini 26.7 → 29.2 tok/s. Prefer v2.

This card is kept for provenance.

Qwen 3.6 35B-A3B · REAP-48 · Q3K-mixed (imatrix)

A 9.4 GB mixture-of-experts coding/tool-use specialist derived from Qwen/Qwen3.6-35B-A3B by pruning 48% of routed experts with REAP and quantizing the result with an attention-aware mixed-precision scheme calibrated by llama-imatrix.

The result fits a 16 GB Mac Mini (M4) — the unmodified base model does not fit on the same hardware at the same quantization scheme — and on coding benchmarks it outperforms a same-quant version of the unmodified base model on three of four metrics.

TL;DR

Method

Compression: REAP, 48 % expert pruning

Routed experts are scored by router-weighted activation magnitude on a 512-sample agentic-coding calibration mix (~1.04 M tokens, seed=42); the lowest-scoring 48 % (123 of 256 routed experts per layer) are pruned, leaving 133. The shared expert and attention layers are preserved. See arXiv:2510.13999 for the method.

Plain REAP is used (no routing-aware variant), because Qwen 3.6 has a shared expert that absorbs ~69 % of layer output norm — it already catches the load when a routed expert is pruned, so the routing-disruption penalty is double-counted by the architecture itself.

Result before quantization: 34.66 B → 19.17 B parameters; bf16 size 41.4 GB.

Quantization: Q3K-mixed + imatrix

Selective scheme via llama-quantize:

Importance matrix computed by llama-imatrix over 1.86 M tokens (253 samples, n_ctx=4096) on the bf16 GGUF, using the same calibration mix as the REAP scoring.

Result: 9.4 GB on disk, 3.93 BPW.

Calibration data

Roughly 60 % agentic / tool-use, 25 % code reasoning, 10 % math, 5 % general:

All datasets retain role structure through the chat template (no flattening).

Inference

llama.cpp / llama-server (recommended)

llama-server \
  -m qwen36-reap-48pct-mixed-q3k.gguf \
  --port 8093 --host 0.0.0.0 \
  -c 8192 -ngl 999 \
  --jinja --reasoning off

Recommended sampling

• temperature: 0 (deterministic for code), or 0.2 for variation
• max_tokens: 2048 for code generation, 4096+ for agentic tool-call chains

KV cache for long context

On a 16 GB Mac, with -ctk q8_0 -ctv q8_0 you can fit the full 262 K native context window. Without KV quantization, expect ~150 K context comfortably.

Performance (M4 Mac Mini, Metal)

Known issues

BigCodeBench-Complete shows a real compression+quant tax

Of all benchmarks measured, BigCodeBench shows the largest gap between this artifact and the unmodified bf16 base: 51.8 % (REAP-48 Q3K) vs 56.3 % (base bf16) — a combined compression+quantization tax of ~4.5 pp, equivalent to ~92 % of base performance retained. The other benchmarks (HumanEval, MBPP+, BFCL-simple) showed taxes well under 1 pp because they are largely saturated. BigCodeBench exercises real-library multi-step reasoning (matplotlib, pandas, sklearn, scipy, requests, etc.) where pruned experts contribute marginally; that's where the cost of removing 48 % of routed experts shows up. The shipping artifact still lands in the upper end of the "Solid" band of the public BigCodeBench-Complete leaderboard, but is not frontier-tier.

MBPP+ small regression vs base Q3K

At Q3K-mixed, REAP-48 loses 11 problems on MBPP+ vs same-quant base (73.5 % vs 76.5 %). MBPP+ specifically stresses Python edge cases (empty inputs, large inputs, type coercion) on small functions; this is the one metric where the compressed model is consistently weaker than the base at this quant level.

Verbose self-correction on the hardest algorithmic problems

On at least one LeetCode-hard problem (median of two sorted arrays in O(log(min(m,n)))) we observed the model write an incomplete attempt, abandon it mid-function with pass, and re-implement the solution in a second function — producing a correct final answer at ~1000 output tokens instead of the ~500 a clean implementation would need. We have not characterized how common this is; if you stream code generation for hard algorithmic prompts, consider max_tokens >= 1500 to allow space for the recovery.

Reproducibility

This artifact was produced by the Crucible compression toolkit (developed alongside this model; source not yet publicly released — planned). Pipeline:

# 1. REAP scoring (on the bf16 base)
crucible observe Qwen/Qwen3.6-35B-A3B \
  --samples 512 --batch-size 2 --seed 42 \
  -o results/qwen36-base-observation.json

# 2. REAP pruning at 48 %
crucible compress Qwen/Qwen3.6-35B-A3B \
  --method reap --ratio 0.48 \
  --scores-file results/qwen36-base-observation.json \
  -o outputs/qwen36-reap-48pct

# 3. f16 GGUF
python llama.cpp/convert_hf_to_gguf.py \
  outputs/qwen36-reap-48pct/Qwen3.6-35B-A3B-reap-48pct \
  --outfile qwen36-reap-48pct-f16.gguf --outtype f16

# 4. imatrix
python crucible/scripts/gen_imatrix_data.py \
  -o /tmp/imatrix_calibration.txt --samples 256
llama-imatrix \
  -m qwen36-reap-48pct-f16.gguf \
  -f /tmp/imatrix_calibration.txt \
  -o qwen36-48pct.imatrix \
  --n-gpu-layers 999 -c 4096

# 5. Q3K-mixed quantize
llama-quantize \
  --imatrix qwen36-48pct.imatrix \
  --token-embedding-type Q8_0 --output-tensor-type Q8_0 \
  --tensor-type attn_q=Q8_0 --tensor-type attn_k=Q8_0 \
  --tensor-type attn_v=Q8_0 --tensor-type attn_output=Q8_0 \
  qwen36-reap-48pct-f16.gguf \
  qwen36-reap-48pct-mixed-q3k.gguf \
  Q3_K_S

Evaluation methodology

All numbers above were produced by Crucible's own-harness bench subcommand (no evalplus / lm-eval runtime dependency). Each problem is run once at temperature=0. Generated code is executed in a subprocess with a 30 s timeout (60 s for BigCodeBench, which has heavier tests). For HumanEval/+ the harness prepends the original prompt to the model's completion (preserving imports). For MBPP+ the assertion list is appended to the model's function. For BFCL-simple, both native tool_calls and in-content <tool_call> / JSON / {fn: {args}} fallbacks are parsed. For BigCodeBench (v0.1.4 split, 1140 problems) the model's body completion is appended to the provided prompt; a unittest.TestLoader-based shim discovers any TestCase subclasses in the assembled module and runs them via TextTestRunner, exiting non-zero on failure. The harness has no dependency on the BigCodeBench evaluator package; it does require the libraries used by the problems themselves (pandas, numpy, matplotlib, seaborn, scikit-learn, scipy, requests, Pillow, etc.) to be present in the runtime, or about half the problems fail at import time.

Note on BFCL on llama.cpp: the BFCL dataset uses Python type names (dict, float, tuple, any) in its tool parameter schemas. vLLM silently accepts these; llama.cpp's --jinja schema validator rejects non-standard types with HTTP 400. The harness's _bfcl_normalize_fn remaps these to valid JSON-Schema types (object, number, array, string) before sending. Without this normalization, BFCL-simple drops ~14-16 percentage points on llama.cpp regardless of model — a runtime fixture issue, not a model regression.

Per-problem result JSONs (task_id / passed / error / raw response, truncated) are retained locally and can be shared on request.

Intended use & limitations

Designed as a coding/tool-use specialist for local single-user inference on consumer hardware. Strong at: function-level Python, idiomatic API usage, multi-step refactors, single- and multi-turn function calling.

Not designed for, and not measured on: multilingual generation, image/audio input, long-form prose, factual QA. General knowledge benchmarks (MMLU, GPQA) were not run on this artifact.

The base Qwen 3.6-35B-A3B model includes a model.visual.* vision encoder (~0.89 GB at bf16) and a multi-token-prediction head (mtp.*, ~1.69 GB). The Crucible compressor operates only on the language-model component (model.language_model.*); these other components are not carried forward into the compressed artifact.

License

Apache-2.0, inherited from Qwen/Qwen3.6-35B-A3B.

Citation

@article{lasby2025reap,
  title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and
          Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
  journal={arXiv preprint arXiv:2510.13999},
  year={2025}
}