SmolLM-135M-Instruct — Q4_0 GGUF

GGUF conversion of HuggingFaceTB/SmolLM-135M-Instruct, a 135M-parameter instruction-tuned model from Hugging Face's SmolLM series.

At ~87 MiB this is about as small as a working chat model gets: it loads in under a second, runs inside a browser tab, and needs almost no RAM. Perfect for smoke tests, demos, edge devices, and teaching how inference works — not for production answers.

Files

• SHA-256: eb2c714d40d4b35ba4b8ee98475a06d51d8080a17d2d2a75a23665985c739b94
• Architecture: Llama (llama)
• Parameters: ~135M (30 layers, hidden 576, 9 heads / 3 KV heads, FFN 1536, head dim 64)
• Context length: 2,048 tokens
• Vocabulary: 49,152 (GPT-2 BPE)
• Chat template: ChatML (<|im_start|> / <|im_end|>) — embedded in the GGUF
• GGUF version: 3

Usage

llama.cpp

llama-cli -m SmolLM-135M-Instruct-Q4_0.gguf --ctx-size 2048 -cnv

Ollama

ollama run hf.co/SadikNawazAI/SmolLM-135M-Instruct-GGUF:Q4_0

Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SadikNawazAI/SmolLM-135M-Instruct-GGUF",
    filename="SmolLM-135M-Instruct-Q4_0.gguf",
    n_ctx=2048,
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=64,
    temperature=0.6,
)
print(out["choices"][0]["message"]["content"])

Recommended sampling

temperature 0.6–0.8, top_p 0.9, repeat_penalty 1.1. Keep prompts short — the model was trained on short single-turn and multi-turn English conversations and degrades quickly with long context.

Provenance

This is Ollama's own GGUF build, taken directly from the smollm:135m manifest (application/vnd.ollama.image.model layer). Other Q4_0 conversions of the same model exist and differ by a few hundred bytes; this one is not a byte-identical copy of them.

License

Apache License 2.0 — see LICENSE. Model weights © Hugging Face (SmolLM team).

Limitations

• 135M parameters: very limited knowledge, frequent hallucinations, weak arithmetic.
• English only, short context (2k), no tool calling.
• Best treated as a toy / test model rather than an assistant you rely on.
• For a usable small chat model at a similar size, prefer Qwen2.5-0.5B-Instruct (0.37 GiB).