bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF

GGUF / llama.cpp quantizations of bottlecapai/ThinkingCap-Qwen3.8-Flash-Next — the ThinkingCap finetune of Qwen3.8-Flash-Next that keeps the base model's answer quality while reasoning in far fewer tokens.

➡️ Full model description, evaluation results (multi-seed, statistically tested), the measured thinking-token reduction, recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.8-Flash-Next.

About GGUF and quantization

GGUF is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, …). The quantized variants below store weights at reduced precision — e.g. about 5 bits per weight for Q4_K_M instead of the unquantized source's 16 — cutting download size and memory severalfold at a small quality cost.

The low-bit files are built with an importance matrix (activation statistics from a calibration corpus, which steer where the low-bit error lands) and per-tensor precision overrides (chosen tensors quantized at a precision other than the quant type's default). Both are taken from unsloth/Qwen3.8-Flash-Next-GGUF.

Files

BF16 is the unquantized source; Q8_0 is near-lossless; Q6_K is a smaller near-lossless option; Q4_K_M is the recommended size/quality balance for most local setups; IQ4_XS is a smaller 4-bit option.

Usage (llama.cpp)

# pull a specific quant straight from the Hub and chat (-hf also loads the vision projector; see Vision below)
llama-cli -hf bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF:Q4_K_M --no-mmproj-offload -p "Hi"

# or download one quant and run it
huggingface-cli download bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF --include "ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-*.gguf" --local-dir .
llama-cli -m ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-00001-of-00008.gguf -p "Hi"

Use the sampling settings from the main model card (the base model's recommended thinking-mode settings). Greedy decoding can loop; keep temperature at the recommended value.

Checked with llama.cpp at commit 5f5acfe (2026-10-22): Q4_K_M with the vision projector answers a math request and an image request correctly, on one H200 (CUDA build) and on CPU.

Speculative decoding (MTP)

These GGUFs carry the model's MTP (multi-token-prediction) head, so llama.cpp can run self-speculative decoding — no separate draft model needed. Add --spec-type draft-mtp when serving:

llama-server -hf bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF:Q4_K_M --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 4

Requires a llama.cpp build with MTP support for this architecture; runtimes that predate it may refuse to load the file with a missing tensor error — update the runtime. In that check on the H200, with 3 draft tokens (--spec-draft-n-max 3), MTP accepted on average 3.75 tokens per decoding step on the math request and 2.72 on the image request.

Vision (image input)

ThinkingCap is a vision-language model. Image input needs the multimodal projector mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf (in this repo) loaded alongside a text GGUF — the single f16 mmproj pairs with any of the quants above.

• LM Studio / Jan / Ollama, …: download the mmproj-*.gguf from this repo; LM Studio auto-detects it and enables the image (🖼️) button.
• llama.cpp CLI:

huggingface-cli download bottlecapai/ThinkingCap-Qwen3.8-Flash-Next-GGUF \
  --include "ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-*.gguf" mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf --local-dir .
llama-mtmd-cli -m ThinkingCap-Qwen3.8-Flash-Next-Q4_K_M-00001-of-00008.gguf \
  --mmproj mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf --no-mmproj-offload --image photo.jpg -p "Describe this image."

• llama-server: add --mmproj mmproj-ThinkingCap-Qwen3.8-Flash-Next-f16.gguf --no-mmproj-offload to expose an OpenAI-compatible vision endpoint.

--no-mmproj-offload runs the vision encoder on the CPU. On CUDA, llama.cpp's flash attention in the vision encoder distorts this model's image embeddings (about 6–9% relative error per image token, the effect reported in llama.cpp#29629), and that costs accuracy: on 200 RealWorldQA questions with IQ4_XS on one H200 (llama.cpp 4e910a8), the GPU encoder scored 70.0% and the CPU encoder 87.5%, against 85.5% for the BF16 model served by vLLM. On the CPU the encoder needs about 1.1 GB of system RAM (and frees as much GPU memory) and about 20 s per 1448 × 938 image on 8 threads, instead of 0.25 s on the GPU.

License

Released under the base model's own license, the Qwen Community License 1.0 (full text in LICENSE). Its conditions carry over to this model, including the separate license from Qwen that Model-as-a-Service and AI work-assistant businesses need for commercial use.