ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

GGUF / llama.cpp files of bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B, our shorter-thinking finetune of NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.

Pre-release. These files have been checked on a four-benchmark evaluation (below), not on the full evaluation behind the main card. No accuracy difference was detected, but the sample is too small to show either file lossless.

➡️ Model description, the full evaluation of the bf16 model, recommended sampling and citation: see the main model card at bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.

Files

f16 is the unquantized conversion, the llama.cpp reference in the evaluation below. Both files carry the multi-token-prediction (MTP) block as blk.52; self-speculative decoding with it is unmeasured on this architecture.

Why Q4_K_M is 6.2 bits per weight here. llama.cpp's k-quants pack weights in blocks of 256 along a row. This model's hidden size (2,688) and expert width (1,856) are not multiples of 256, so llama.cpp falls back to Q5_0 and Q8_0 for those tensors: the expert up-projections are all Q5_0 and the expert down-projections 54% Q5_0 and 46% Q8_0. Only the projections whose input width is a multiple of 256 (attention output and Mamba-2 output, 4,096 wide, and the MTP input projection) get Q4_K. The file is therefore larger, and closer to the f16, than "4-bit" suggests.

How it was built. llama.cpp commit bfd73a8: convert_hf_to_gguf.py to f16 with the MTP block kept, an importance matrix from bartowski's calibration v6 corpus (583 chunks × 512 tokens, chat special tokens parsed), then llama-quantize Q4_K_M with that matrix and llama.cpp's default per-tensor rules. Fidelity to the f16 on wiki.test (100 chunks × 512 tokens): mean KL divergence 0.0188, same top token 94.0% of the time.

Usage (llama.cpp)

# pull a quant straight from the Hub and chat (this repo is private: export HF_TOKEN first)
llama-cli -hf bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_K_M

# or serve an OpenAI-compatible endpoint
llama-server -hf bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_K_M --jinja

The file embeds the model's chat template; thinking is on by default. Use NVIDIA's recommended sampling, temperature=1.0, top_p=0.95; greedy decoding can loop.

Tested with llama.cpp bfd73a8 (CUDA, H200). Not yet tested: Ollama, LM Studio, Metal, Vulkan and CPU-only builds.

Expected performance

What was compared. The two GGUF files served by llama.cpp llama-server (commit bfd73a8, f16 KV cache, 16 parallel slots), against the bf16 model served by vLLM 0.31.0 (bf16 KV cache), one H200 each, answering the same questions with the same sampling seeds. Questions are fixed subsets: GPQA-Diamond 60 of 198 questions × 2 seeds, MMLU-Pro 100 of 12,032 × 1, IFBench 40 of 300 × 1, AA-LCR 20 of 100 × 1 (long-document questions; answers graded by Gemma-4-26B-A4B-it with thinking off). Thinking on, NVIDIA's recommended sampling (temperature 1.0, top_p 0.95), 65,536-token generation cap, no speculative decoding.

Q4_K_M against f16 changes only the weights (same engine, same GPU type). f16 against bf16 changes the engine and the 16-bit format with no low-bit quantization — the engine control.

Metrics. Accuracy: the share of question × seed answers scored correct. Δ accuracy: difference in percentage points, with a 95% bootstrap interval that resamples questions (seeds averaged per question first). Thinking tokens: length of the reasoning trace, answer excluded, over every question × seed; p95 is its 95th percentile. Δ mean thinking tokens: one mean over the other on the same questions, minus one, with a question-bootstrap interval.

Accuracy

Thinking tokens

Does the shorter thinking survive Q4_K_M? ThinkingCap against the base model (NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) converted and quantized with the same recipe, at the same precision and on the same engine. Change in mean thinking tokens [95% CI]; the accuracy change against base at Q4_K_M is in the last column, in pp.

• Accuracy: no difference was detected for either file; but Q4_K_M's intervals against f16 still allow losses of up to 7.5 pp on GPQA-Diamond, 8 pp on MMLU-Pro and 15 pp on AA-LCR.
• Thinking length: no change from Q4_K_M against f16 is detected. Against the base model at Q4_K_M, ThinkingCap thinks 28% less on GPQA-Diamond and 29% less on MMLU-Pro and 33% less on IFBench (interval reaches +0.4%); on AA-LCR the reduction is 9% on 20 questions, with an interval spanning zero.
• The absolute numbers differ from the main card's, which uses the full question sets, more seeds and an FP8 KV cache on vLLM; compare within these tables only.

Where to find us

Need even more efficiency? Our enterprise versions go further — fewer thinking tokens still, tuned to your workload, at matched accuracy on your own tasks. Deployed on your infrastructure, or in the cloud and region you choose. Talk to our team

License

Released under the base model's own license, the OpenMDW License Agreement 1.1 (full text in LICENSE).

Citation

If you use this model, please cite:

@misc{ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B,
  title     = {bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B},
  author    = {Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Kubista, Daniel and Krus, Frantisek and Mikolov, Tomas},
  year      = {2026},
}