Inkling 2 Mini for Basalt
Website: basalt.jgulfo.com
Model file for Basalt, a local inference engine for NVIDIA Blackwell GPUs (sm_120) with an OpenAI- and Anthropic-compatible API. Basalt runs this model on one Blackwell card, with the least used experts in system RAM only when a long context needs the card's room.
One self-contained file: weights, the vision encoder, the tokenizer and chat template and the DSpark drafter, built from Thinking Machines Lab's official release (thinkingmachines/Inkling-2-Mini). Needs Basalt 0.5.5 or later.
| file | size | experts |
|---|---|---|
Inkling-2-Mini-NVFP4-q8.basalt |
19.7 GB | the release's own: NVFP4 (layers 0-19), MXFP8 as Q8_0 (layers 20-23) |
SHA-256: f7b4e5f8e17681c4735a7dfc51ffd8e805c44184fad4e1778b13187d79dc3787
- Vision: images work with
--vision dynamic, encoded inside the engine; a picture is read at its own size, one token per 40 x 40 pixel cell. - Context: up to 1,048,576 tokens (
--max-context).--kv fp8halves the cache. When the asked context does not fit beside every expert, the least routed experts move to RAM, just enough, and come back to the card while the cache leaves room. - Drafter: DSpark drafts by default (up to seven tokens a pass). Drafted tokens are verified, so the output is the model's own.
--spec 1turns it off. - Thinking and tools: the chat template's reasoning effort levels and tool calls work through the server.
What is in the file
The routed experts are Thinking Machines' own quantization, moved over without rounding again; the rest is rounded once.
| part | format |
|---|---|
| Routed experts, layers 0-19 (128 per layer, top 8) | NVFP4 codes and scales, bit for bit from the release |
| Routed experts, layers 20-23 | the release's MXFP8 values as Q8_0 |
| Attention, gate, short-convolution and output projections | Q8_0 |
| Output head | BF16 (the engine rounds it to Q8_0 at load) |
| Token embedding | BF16 |
| Norms, router, biases, loop's small tensors | F32 (loop controllers BF16) |
| Vision encoder and projector | BF16 |
| DSpark drafter | BF16 |
Speed
Single stream, one RTX 5090 at 400 W, every expert on the card, basalt-bench (seeded, two runs), tokens/s.
| conversation | structured | prose | prefill, 16K prompt |
|---|---|---|---|
| 338 | 613 | 287 | 12,240 |
DSpark drafting on (the default). Prompts of 2K, 32K and 128K tokens read at about 14,100, 12,200 and 8,900 tok/s. At
--max-context 1048576 --kv fp8 (experts spilled to RAM as needed): 264 tok/s on a short chat, 173 at 131K and 108 at 400K
(plain decoding); a 400K prompt reads in about 97 s. Needles at 131K and 400K are found.
Quality
Measured with basalt-bench --quality through the decode path against the official release with its experts dequantized to
BF16 (the release ships no BF16 weights, so this reference is not a true BF16 model and its KLD is not comparable with other
models'). Wikitext-2, 512-token chunks.
| chunks | PPL | reference PPL | KLD mean | same top-1 |
|---|---|---|---|---|
| 100 | 7.0695 | 7.0633 | 0.00305 | 97.76% |
| 20 | 7.6642 | 7.6534 | 0.00264 | 97.80% |
Run
Build the engine as described in the Basalt README, then:
hf download jesdga/Inkling-2-Mini-Basalt Inkling-2-Mini-NVFP4-q8.basalt --local-dir models
bin/basalt serve models/Inkling-2-Mini-NVFP4-q8.basalt --vision dynamic
Add --max-context N for a longer context. More in
Inkling 2 Mini on Basalt and
models and the router.
Sources
- Model: thinkingmachines/Inkling-2-Mini, Thinking Machines Lab.
- This pack: built from the official release by
tools/convert/convert.py --checkpoint; tokenizer and chat template from it. - Basalt is a fork of Strata (MIT).
License
The model is released by Thinking Machines Lab under Apache-2.0. This file is a modified (quantized and repacked) form of the original weights. This is not an official Thinking Machines release. The Basalt engine is MIT.