Qwen3-4B-Instruct-2507-heretic - GGUF
GGUF quantizations of heretic-org/Qwen3-4B-Instruct-2507-heretic, produced with llama.cpp.
Available files
| File | Quantization |
|---|---|
| model-F16.gguf | F16 |
| model-Q8_0.gguf | Q8_0 |
| model-Q6_K.gguf | Q6_K |
| model-Q5_K_M.gguf | Q5_K_M |
| model-Q5_K_S.gguf | Q5_K_S |
| model-Q5_0.gguf | Q5_0 |
| model-Q4_K_M.gguf | Q4_K_M |
| model-Q4_K_S.gguf | Q4_K_S |
| model-Q4_0.gguf | Q4_0 |
| model-Q3_K_L.gguf | Q3_K_L |
| model-Q3_K_M.gguf | Q3_K_M |
| model-Q3_K_S.gguf | Q3_K_S |
| model-Q2_K.gguf | Q2_K |
Usage
With llama.cpp:
./llama-cli -m model-Q4_K_M.gguf -p "Your prompt" -n 256
License
Apache 2.0, inherited from the base model.