Qwen3-4B Q4_K_M GGUF
This is a Q4_K_M quantized GGUF conversion of Qwen/Qwen3-4B optimized for on-device inference with llama.cpp.
Model Details
| Property | Value |
|---|---|
| Original Model | Qwen3-4B |
| Parameters | 4 billion |
| Quantization | Q4_K_M (4-bit, medium quality) |
| File Size | ~2.5 GB |
| Context Window | 32,768 tokens |
| Architecture | Qwen3 (RoPE, SwiGLU, RMSNorm, GQA) |
Intended Use
This model is optimized for:
- Mobile/Edge Deployment: Runs efficiently on iOS devices with 8GB+ RAM (iPhone 15 Pro, 16, etc.)
- llama.cpp Integration: Compatible with llama.cpp and its bindings
- On-Device AI: Private, offline inference without cloud dependencies
Capabilities
- Complex reasoning and analysis
- Code generation and debugging
- Mathematical problem solving
- Creative writing
- Multilingual support (29+ languages)
- "Thinking mode" for step-by-step reasoning
Usage with llama.cpp
./llama-cli -m Qwen3-4B-Q4_K_M.gguf -p "Your prompt here" -n 512
License
This model inherits the Apache 2.0 license from the original Qwen3-4B model.
Attribution
- Original Model: Qwen3-4B by Qwen Team, Alibaba Cloud
- Quantization: jc-builds