Huihui-MiniCPM5-2B-abliterated — GGUF Quantizations
This repository contains GGUF quantizations of huihui-ai/Huihui-MiniCPM5-2B-abliterated.
📦 Available Quantizations
| File | Bits | Size (approx.) | Use case |
|---|---|---|---|
model_f16.gguf |
16-bit | ~5.0 GB | Maximum quality, reference |
model_q8_0.gguf |
8-bit | ~2.7 GB | Near-lossless, high VRAM |
model_q6_k.gguf |
6-bit | ~2.1 GB | Excellent quality |
model_q5_k_m.gguf |
5-bit | ~1.8 GB | Great quality/size balance |
model_q5_k_s.gguf |
5-bit | ~1.7 GB | Slightly smaller than K_M |
model_q4_k_m.gguf |
4-bit | ~1.5 GB | Recommended default ✅ |
model_q4_k_s.gguf |
4-bit | ~1.4 GB | Smaller 4-bit variant |
model_q3_k_l.gguf |
3-bit | ~1.2 GB | Low VRAM, decent quality |
model_q3_k_m.gguf |
3-bit | ~1.1 GB | Balanced 3-bit |
model_q3_k_s.gguf |
3-bit | ~1.0 GB | Minimum 3-bit |
model_q2_k.gguf |
2-bit | ~0.8 GB | Extreme compression |
🚀 Usage
llama.cpp
./llama-cli -m model_q4_k_m.gguf -p "Your prompt here" -n 512
LM Studio
Download any .gguf file and load it directly in LM Studio.
Ollama
ollama run hf.co/tinyopsec/Huihui-MiniCPM5-2B-abliterated-GGUF:Q4_K_M
Python (llama-cpp-python)
from llama_cpp import Llama
llm = Llama.from_pretrained(
repo_id="tinyopsec/Huihui-MiniCPM5-2B-abliterated-GGUF",
filename="model_q4_k_m.gguf",
)
output = llm("Your prompt here", max_tokens=512)
print(output["choices"][0]["text"])
🔧 Quantization Details
- Source model: huihui-ai/Huihui-MiniCPM5-2B-abliterated
- Tool: llama.cpp
- Base format: F16 GGUF
💡 Which quant should I use?
| VRAM | Recommended |
|---|---|
| 1 GB | Q2_K |
| 2 GB | Q3_K_M |
| 3 GB | Q4_K_M ✅ |
| 4 GB | Q5_K_M |
| 6 GB | Q6_K |
| 8 GB+ | Q8_0 / F16 |
📄 License
Refer to the original model license.