SmolLM2-1.7B-Instruct Q4_K_M GGUF
This is a Q4_K_M quantized GGUF conversion of HuggingFaceTB/SmolLM2-1.7B-Instruct optimized for on-device inference with llama.cpp.
Model Details
| Property | Value |
|---|---|
| Original Model | SmolLM2-1.7B-Instruct |
| Parameters | 1.7 billion |
| Quantization | Q4_K_M (4-bit, medium quality) |
| File Size | ~1.0 GB |
| Context Window | 8,192 tokens |
| Architecture | LLaMA-style transformer |
| Training Data | 11 trillion tokens |
Intended Use
This model is optimized for:
- Mobile/Edge Deployment: Runs efficiently on most iOS devices
- llama.cpp Integration: Compatible with llama.cpp and its bindings
- On-Device AI: Private, offline inference without cloud dependencies
Capabilities
- Best in SmolLM2 Family: Highest quality from the SmolLM2 series
- Function Calling: Supports tool use and function calling
- Instruction Following: Excellent at following complex instructions
- Code Generation: Good coding capabilities
- Efficient: Trained on 11T tokens for maximum efficiency
Usage with llama.cpp
./llama-cli -m SmolLM2-1.7B-Instruct.Q4_K_M.gguf -p "Your prompt here" -n 512
License
This model inherits the Apache 2.0 license from the original SmolLM2 model.
Attribution
- Original Model: SmolLM2-1.7B-Instruct by Hugging Face
- Quantization: jc-builds