Inkling-Small-MLX-REAP25-4bit

Built with Inkling (Thinking Machines Lab).

A REAP-pruned, 4-bit MLX build of thinkingmachines/Inkling-Small: each MoE layer keeps its 192 highest-saliency routed experts (of 256), a 25% expert prune. No measurable loss — the build to take for a 128 gb mac.

Code / loader: github.com/PipeNetwork/inkling-mlx

What is REAP pruning?

REAP (Router-weighted Expert Activation Pruning, Cerebras, arXiv:2510.13999) ranks each routed expert by saliency = mean over the tokens that route to it of router_gate_weight × ‖expert_output‖₂ — its actual contribution to the residual stream. The lowest-saliency experts are dropped; the router simply renormalizes over the survivors (no weight surgery). The 2 shared "sink" experts, attention, and embeddings are untouched. Inkling routes very uniformly (routing entropy 0.908; only ~0.15 cold experts per layer once audio is included in the calibration), so it is only lightly prunable — reflected below.

Calibrated on text, images and audio (this matters)

Inkling-Small is multimodal, and expert saliency was profiled over a mixed corpus of text (code + 15 languages + reasoning), 200 real images, and 180 speech clips run through the full vision and audio paths. On this model that is not optional: 47.7 experts per layer are >50% audio-driven and 22.3 are >50% image-driven — about 27% of all experts serve primarily non-text input, and adding audio to the calibration dropped the cold-expert count from 0.65 to 0.15 per layer. Experts that only ever fire on speech look worthless to a text-only profiler and get pruned first, which is how the 975B model lost speech transcription (word-overlap 0.88 → 0.57) while its text perplexity still looked fine. Profiling all three modalities keeps them. On held-out tests this build scores vision 6/6 and audio 0.896 word-overlap — both at the unpruned build's level.

Measured quality (4-bit)

This build: text perplexity 4.992 (−8.4% vs the unpruned 4-bit), vision 6/6 (held-out image ID), audio 0.896 (held-out speech transcription word-overlap), 87.5% of router-weighted expert contribution retained.

Read that −8.4% as "no measurable change", not as pruning improving the model. Perplexity was measured on two independent held-out sets: this one has REAP-25 at −8.4% vs the unpruned 4-bit, a second one has it at +0.55%. Sets that small disagree by a few percent in either direction, so the defensible claim is that a 25% expert prune costs nothing measurable here — not that it helps.

50% pruning is a different story and is deliberately not published. It was built and evaluated: text perplexity roughly doubled (+88% / +105% on the two sets) and speech transcription fell from 0.874 to 0.702, so it was dropped rather than shipped with a warning. Inkling-Small is meaningfully less prunable than the 975B model, whose REAP-50 cost ~22%. Pruning is applied to the already-quantized build; because expert subsetting is along the expert axis and affine-quant groups run along the hidden axis, it is bit-identical to pruning the bf16 source then requantizing.

Quantization scheme: affine int4 (not NVFP4 / MXFP4)

MLX supports FP4 modes and Thinking Machines ships an Inkling-NVFP4 checkpoint — so for the record, we benchmarked round-trip reconstruction error (‖W − Ŵ‖ / ‖W‖ vs bf16) on real Inkling expert weights:

Affine int4 is the most faithful: it is asymmetric (per-group scale and zero-point, 16 uniform levels), which centers on Inkling's near-Gaussian expert weights better than symmetric FP4's fixed non-uniform levels (scale only, no zero-point). FP4's real payoff is heavy-tailed activations and native Blackwell FP4 tensor cores — neither helps weight fidelity on Apple Silicon, where MLX would dequantize FP4 anyway. So these builds use affine int4; a Mac port of the NVFP4 checkpoint would be lower quality at best-equal size.

⚠️ Loading requires the bundled inkling_mlx loader

The inkling_mm_model architecture is not in stock mlx-lm / mlx-vlm, so this repo bundles a minimal, numerically-validated MLX implementation under inkling_mlx/. The reduced expert count is recorded in config.json (n_routed_experts = 192) and the loader builds the model to match automatically.

pip install mlx mlx-lm transformers

from inkling_mlx.load import load
from inkling_mlx.generate import greedy_generate
from transformers import AutoTokenizer

model, config = load("/path/to/this/repo")            # eager wired load fits comfortably
tok = AutoTokenizer.from_pretrained("/path/to/this/repo", trust_remote_code=True)
ids = tok("The capital of France is")["input_ids"]
print(tok.decode(greedy_generate(model, config, ids, max_new_tokens=64)))

Needs an Apple-Silicon Mac with unified memory ≥ the size above. The smaller footprint is the practical point: ~112 GB fits a 128 GB Mac, where the unpruned 148 GB 4-bit build does not.

Details

• Multimodal (HMLP vision + dMel audio towers + preprocessing) is included, same as the base MLX build; the multi-token-prediction head is dropped.
• Quantized: attention / MLP / expert projections, embed+unembed, vision/audio matmuls. Kept higher precision: MoE router, RMSNorms, the four short-convolutions per layer, relative-position bias.

License: Apache-2.0 (inherits the base model).