Kimi-VL-A3B-Thinking GGUF
GGUF quantizations of moonshotai/Kimi-VL-A3B-Thinking-2506 for use with llama.cpp and Ollama.
Model Description
Kimi-VL-A3B-Thinking is a powerful vision-language model with extended thinking capabilities from Moonshot AI. It features a Mixture of Experts (MoE) architecture built on DeepSeek2 for efficient inference with strong reasoning capabilities.
Key Features
- Vision & Reasoning - Understands images and uses chain-of-thought reasoning
- 128K Context - Massive 131,072 token context window
- MoE Architecture - 64 experts + 2 shared experts for efficient inference
- DeepSeek2 Base - Built on proven DeepSeek2 architecture with MLA attention
Available Quantizations
| Filename | Quant | Size | Description |
|---|---|---|---|
| Kimi-VL-A3B-Thinking-Q4_K_M.gguf | Q4_K_M | 9.8 GB | Best balance of quality and speed (recommended) |
| Kimi-VL-A3B-Thinking.gguf | F16 | 30 GB | Full precision |
Usage
With Ollama
# Pull and run (Q4_K_M by default)
ollama run richardyoung/kimi-vl-a3b-thinking
# Or specific quantization
ollama run richardyoung/kimi-vl-a3b-thinking:f16
With llama.cpp
# Download a quantization
wget https://huggingface.co/richardyoung/Kimi-VL-A3B-Thinking-GGUF/resolve/main/Kimi-VL-A3B-Thinking-Q4_K_M.gguf
# Run with llama.cpp
./llama-cli -m Kimi-VL-A3B-Thinking-Q4_K_M.gguf -p "Analyze this image step by step:" --image your_image.jpg
Technical Requirements
- Minimum: 16GB RAM
- Recommended: 32GB RAM or Apple Silicon Mac with 24GB+ unified memory
Chat Template
Kimi-VL uses a custom template format:
<|im_system|>system<|im_middle|>{system_message}<|im_end|>
<|im_user|>user<|im_middle|>{user_message}<|im_end|>
<|im_assistant|>assistant<|im_middle|>{assistant_response}<|im_end|>
Links
- Original Model: moonshotai/Kimi-VL-A3B-Thinking-2506
- Ollama: richardyoung/kimi-vl-a3b-thinking
Credits
- Original Model: Moonshot AI
- Quantization: Richard Young (deepneuro.ai)
License
MIT License
Built & maintained by Richard Young · DeepNeuro