A tiny yet powerful instruction-tuned language model optimized for CPU inference. With only 135 million parameters and a file size of 138 MB, this model delivers impressive performance even on modest hardware.
smollm2-135m-instruct-q8_0.gguf - Main model file (Q8_0 quantized)tokenizer.json - Model tokenizer fileconfig.json - HuggingFace compatibility configurationLICENSE - Apache 2.0 license fileREADME.md - This documentation# Install llama-cpp-python
pip install llama-cpp-python
# Basic usage
./main -m smollm2-135m-instruct-q8_0.gguf -p "Who are you?"
# With custom parameters
./main -m smollm2-135m-instruct-q8_0.gguf --ctx-size 2048 --threads 4 -p "Write a story."
from llama_cpp import Llama
# Initialize the model
llm = Llama(
model_path="smollm2-135m-instruct-q8_0.gguf",
n_ctx=2048, # Context window
n_threads=4, # CPU threads to use
n_batch=512 # Batch size for prompt processing
)
# Generate a response
output = llm("What is the capital of France?",
max_tokens=128,
temperature=0.7,
top_p=0.95)
print(output)
This is a chat-style instruction-tuned model. Use the following message format for best results:
[
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "Tell me a joke."}
]
User: What is your name?
Luna: My name is Luna, and I'm your tiny but capable AI assistant, ready to help with anything you need!
SmolLM2 models primarily understand and generate content in English. They can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content.
Feel free to Like ❤️ the repository if you find this model useful!