Inkling-Small-REAP-137B-A12B-de (GGUF)
A REAP expert-pruned variant of thinkingmachines/Inkling-Small (276B-A12B → 137B total / 12B active), provided as GGUF quantizations.
Variants in this repo
| Variant | Size | BPW | Notes |
|---|---|---|---|
| IQ4_XS (imatrix) | 72.0 GB | ~4.3 | Recommended — importance-matrix-weighted, smaller and typically better than plain Q4_K_M |
| Q4_K_M | 81.5 GB | 4.84 | Plain k-quant, no imatrix |
imatrix.dat |
217 MB | — | Importance matrix (538 entries) — build your own quants with llama-quantize --imatrix |
The importance matrix was computed on a German-heavy DE/EN mix matching the pruning calibration (German multi-register human text, German model outputs, English code and math), so the quantization error budget follows the same language priorities as the expert pruning. Both variants are split into ≤45 GB parts; keep the parts of one variant in the same directory and load the first part.
The idea: produce a sparse model in the 100–140B class with a moderate quantization level (Q4_K_M, no aggressive sub-4-bit tricks), so that the quality loss budget is spent once on expert pruning and once on a well-understood quantization — not twice on extreme compression. We plan to spot-benchmark it against dense/sparse models of similar deployed size (e.g. Qwen3.5-122B-class models).
⚠️ Language disclaimer: the expert-pruning calibration was optimized for German and English (German-heavy multi-register calibration mix). Losses in other languages are presumably significantly higher — experts that were mostly used by other languages were preferentially removed. If you need multilingual coverage beyond DE/EN, use the original Inkling-Small.
What was done
- Method: REAP (Router-weighted Expert Activation Pruning, Lasby et al., ICLR 2026) — one-shot, no retraining. Per MoE layer, the 128 of 256 routed experts with the lowest router-weighted activation saliency were removed; router rows / e-score bias sliced accordingly. Shared experts, attention, dense layers, embeddings and MTP weights are untouched.
- Calibration: ~4M tokens, 56 % German (human-written text across 10 registers — news, legal, parliamentary, scientific, encyclopedic, spoken, easy language, essays, social, narrative — plus German instruction/chat data and German outputs of frontier models), 44 % English code / reasoning / tool-calling / chat.
- Quantization: llama.cpp Q4_K_M from BF16.
Calibration only collects routing statistics; no calibration text enters the weights.
Internal retention probes (not benchmarks)
Teacher-forced streaming probes against the BF16 base model (ΔNLL in nats, top-1 agreement with base). Spot checks, not standardized benchmarks:
| Probe | ΔNLL vs. base | Top-1 agreement |
|---|---|---|
| German prose | +0.39 | 66.7 % |
| German factual text | −0.18 | 77.2 % |
| English prose | +0.72 | 73.0 % |
| Python | +0.68 | 81.0 % |
For comparison: an English-only-calibrated pruning of the same ratio degraded German prose by +1.67 nats. Proper benchmark results (vs. Qwen3.5-122B-class models) are planned.
Usage
Requires a llama.cpp build with Inkling support — currently the
add-inkling PR #25731
until it is merged:
git clone -b add-inkling https://github.com/danielhanchen/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-cli -m Inkling-Small-REAP-137B-A12B-de-Q4_K_M-00001-of-00002.gguf \
-p "Erkläre den Unterschied zwischen Bundesrat und Bundestag." -n 300
The split GGUF parts must sit in the same directory; llama.cpp loads them automatically from the first part.
Limitations
- DE/EN optimized — other languages likely degrade well beyond typical REAP losses (see disclaimer above).
- Code/tool-calling gives up a little quality relative to an English/code-only calibration (that trade-off was chosen deliberately).
- Text-only: vision/audio towers of the multimodal original are not part of this GGUF.
- No MTP speculative decoding in this GGUF (llama.cpp Inkling MTP support pending). The underlying pruned safetensors checkpoint retains the full MTP stack.
- Pruned models remain probabilistic compressions of the original — verify before production use.
About the base model: Inkling-Small
⚠️ Carry-over disclaimer: Everything in this section describes the original thinkingmachines/Inkling-Small (276B-A12B, BF16, multimodal). This repo contains a 50 %-expert-pruned, Q4_K_M-quantized, text-only derivative — the properties and scores below do not all carry over. Treat every number as an upper bound, not a claim about this model. Multimodal inputs are absent here entirely, and multilingual ability outside German/English was deliberately traded away during pruning calibration.
Key facts (original)
| Property | Original Inkling-Small |
|---|---|
| Model type | Multimodal autoregressive transformer (text/image/audio → text) |
| Architecture | 42-layer decoder, sparse MoE FFN: top-6 of 256 routed experts + 2 shared; hybrid local/global attention |
| Parameters | 276B total / 12B active (this repo: 137B total / 12B active) |
| Context length | up to 1M tokens (long-context behavior of the pruned model is untested) |
| Numerics | BF16 and NVFP4 (this repo: Q4_K_M GGUF) |
| Languages | English, with general multilingual capabilities (this repo: DE/EN-calibrated) |
| Training data | Public, third-party-acquired and synthetic text/image/audio/video (see original card) |
| MTP | 8 speculative multi-token-prediction layers (preserved in our safetensors, not in this GGUF) |
| License | Apache-2.0 |
Selected original benchmark scores
Scores of the unpruned BF16 original, from the official model card (where it is compared against Qwen3.5-397B-A17B, MiMo V2.5, Minimax M2.7 and closed models — see there for the full 10-model table). None of these numbers have been re-measured for this pruned+quantized variant.
| Benchmark | Original Inkling-Small |
|---|---|
| SWEBench Verified | 80.2 % |
| Terminal Bench 2.1 (best harness) | 64.7 % |
| SciCode | 48.7 % |
| GPQA Diamond | 89.5 % |
| AIME 2026 | 95.5 % |
| HLE (text only / with tools) | 31.6 % / 47.8 % |
| ARC-AGI-1 / ARC-AGI-2 | 84.0 % / 40.1 % |
| BrowseComp (with context mgmt) | 77.4 % |
| IFBench | 82.2 % |
| Global-MMLU-Lite | 86.7 % (expect larger drops off DE/EN here) |
| SimpleQA Verified | 20.6 % |
| StrongREJECT (safety) | 98.4 % |
Multimodal scores (MMMU Pro, Charxiv, MMAU, VoiceBench …) are omitted: the vision/audio towers are not part of this GGUF.
Deployment of the original
The original model has day-0 recipes for SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face transformers (links in the original card), including MTP speculative decoding. This GGUF instead targets llama.cpp (see Usage above); our pruned safetensors checkpoint (TML layout, MTP intact, vLLM/SGLang-compatible) is not published here — open a discussion if you need it.
Safety, bias and limitations (inherited)
The original card documents safety training, known biases, risks and legal terms — all of that applies here at best unchanged and plausibly degraded: expert pruning is a lossy intervention whose effect on safety behavior and refusal calibration has not been separately evaluated. Review the original card's Safety / Bias / Legal sections before production use.
Provenance
- Base: thinkingmachines/Inkling-Small (Apache-2.0)
- Pruning: REAP streaming pipeline (calibration ran block-wise on a single RTX 5090 workstation without ever loading the 532 GB model into memory)
- Per-layer pruned expert lists: see
reap_pruning.jsonin this repo
@inproceedings{lasby2026reap,
title={{REAP} the Experts: Why Pruning Prevails for One-Shot MoE compression},
author={Lasby, Lazarevich, Sinnadurai, Lie, Ioannou, Thangarasa},
booktitle={ICLR}, year={2026}
}
Deutsch (Kurzfassung)
REAP-geprunte Variante von Inkling-Small: pro Layer wurden die 128 von 256 Experten mit der geringsten router-gewichteten Aktivierungs-Salienz entfernt (one-shot, kein Training), anschließend Q4_K_M-Quantisierung. Die Kalibrierung war bewusst deutschlastig (56 %, menschliche Texte über zehn Register plus deutsche Instruktionsdaten): deutsche Prosa verliert nur +0,39 nats gegenüber dem Original (statt +1,67 bei englischer Kalibrierung), deutscher Sachtext liegt auf Originalniveau. In anderen Sprachen als Deutsch und Englisch sind die Verluste mutmaßlich deutlich höher. Geplant sind Stichproben-Benchmarks gegen Modelle der Qwen3.5-122B-Klasse.