TinyLlama-1.1B-Chat-v1.0-pruned2.4 GGUF
GGUF quantizations of RedHatAI/TinyLlama-1.1B-Chat-v1.0-pruned2.4, a 2:4 semi-structured sparse version of TinyLlama-1.1B-Chat-v1.0 pruned with SparseGPT via SparseML.
About the Original Model
TinyLlama-1.1B-Chat-v1.0-pruned2.4 is a compressed variant of TinyLlama 1.1B Chat, pruned to 2:4 sparsity (50% sparse) using the SparseGPT one-shot method. The pruning was performed by RedHat AI using SparseML on the open_platypus calibration dataset. The model retains conversational capability while being significantly more hardware-efficient.
Architecture: LlamaForCausalLM
Parameters: ~1.1B
Context length: 2048 tokens
Sparsity: 2:4 semi-structured (50%)
Quantization Files
| File | Bits | Approx Size | Use Case |
|---|---|---|---|
model_f16.gguf |
16 | ~2.2 GB | Maximum quality, reference |
model_q8_0.gguf |
8 | ~1.2 GB | Best quality / size tradeoff |
model_q6_k.gguf |
6 | ~0.9 GB | High quality, smaller |
model_q5_k_m.gguf |
5 | ~0.8 GB | Balanced quality |
model_q5_k_s.gguf |
5 | ~0.75 GB | Slightly smaller Q5 |
model_q4_k_m.gguf |
4 | ~0.67 GB | Good quality, low RAM |
model_q4_k_s.gguf |
4 | ~0.63 GB | Smaller Q4 |
model_q3_k_l.gguf |
3 | ~0.55 GB | Low RAM, acceptable quality |
model_q3_k_m.gguf |
3 | ~0.51 GB | Lower RAM |
model_q3_k_s.gguf |
3 | ~0.47 GB | Minimum RAM Q3 |
model_q2_k.gguf |
2 | ~0.38 GB | Extreme compression |
VRAM Requirements (Estimated)
| Quantization | VRAM |
|---|---|
| F16 | ~2.4 GB |
| Q8_0 | ~1.4 GB |
| Q6_K | ~1.1 GB |
| Q5_K_M | ~0.95 GB |
| Q4_K_M | ~0.85 GB |
| Q3_K_M | ~0.70 GB |
| Q2_K | ~0.55 GB |
Prompt Template
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
Inference
llama.cpp
./llama-cli -m model_q4_k_m.gguf -p "<|im_start|>user\nHello!<|im_end|>\n<|im_start|>assistant\n" -n 256
llama-cpp-python
from llama_cpp import Llama
llm = Llama(model_path="model_q4_k_m.gguf", n_ctx=2048)
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "Hello!"}]
)
print(output["choices"][0]["message"]["content"])
LM Studio
- Open LM Studio
- Search for
tinyopsec/TinyLlama-1.1B-Chat-v1.0-pruned2.4-GGUF - Download the desired quantization
- Load and chat
Ollama
ollama run hf.co/tinyopsec/TinyLlama-1.1B-Chat-v1.0-pruned2.4-GGUF:Q4_K_M
Links
- Original model: RedHatAI/TinyLlama-1.1B-Chat-v1.0-pruned2.4
- Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
- SparseGPT paper: arxiv:2301.00774