OmniCoder-9B-GGUF
GGUF quantizations of OmniCoder-9B
Available Quantizations
| Quantization | Size | Use Case |
|---|---|---|
Q2_K |
~3.8 GB | Extreme compression, lowest quality |
Q3_K_S |
~4.3 GB | Small footprint |
Q3_K_M |
~4.6 GB | Small footprint, balanced |
Q3_K_L |
~4.9 GB | Small footprint, higher quality |
Q4_0 |
~5.3 GB | Good balance |
Q4_K_S |
~5.4 GB | Good balance |
Q4_K_M |
~5.7 GB | Recommended for most users |
Q5_0 |
~6.3 GB | High quality |
Q5_K_S |
~6.3 GB | High quality |
Q5_K_M |
~6.5 GB | High quality, balanced |
Q6_K |
~7.4 GB | Near-lossless |
Q8_0 |
~9.5 GB | Highest quality quantization |
BF16 |
~17.9 GB | Full precision |
Usage
# Install llama.cpp
brew install llama.cpp # macOS
# or build from source: https://github.com/ggml-org/llama.cpp
# Interactive chat
llama-cli --hf-repo Tesslate/OmniCoder-9B-GGUF --hf-file omnicoder-9b-q4_k_m.gguf -p "Your prompt" -c 8192
# Server mode (OpenAI-compatible API)
llama-server --hf-repo Tesslate/OmniCoder-9B-GGUF --hf-file omnicoder-9b-q4_k_m.gguf -c 8192
Built by Tesslate | See full model card: OmniCoder-9B