Bible Assistant Qwen3.5-4B-v2 GGUF
GGUF quantized versions of Ttimms/Bible-Assistant-Qwen3.5-4B-v2.
Quantization Details
| Quantization | Bits | Size | Use Case |
|---|---|---|---|
| Q8_0 | 8 | ~4.2 GB | High quality, minimal loss |
| Q6_K | 6 | ~3.2 GB | Balanced quality/size |
| Q5_K_M | 5 | ~2.7 GB | Good quality, reduced size |
| Q5_K_S | 5 | ~2.5 GB | Faster inference, slight quality loss |
| Q4_K_M | 4 | ~2.1 GB | Recommended default |
| Q4_K_S | 4 | ~1.9 GB | Faster, smaller footprint |
| Q3_K_L | 3 | ~1.6 GB | Mobile-friendly |
| Q3_K_M | 3 | ~1.5 GB | Aggressive quantization |
| Q3_K_S | 3 | ~1.4 GB | Mobile/edge devices |
| Q2_K | 2 | ~1.1 GB | Extreme compression |
VRAM Requirements
| Quantization | VRAM | Device |
|---|---|---|
| Q8_0 | ~4.5 GB | Desktop GPU |
| Q6_K | ~3.5 GB | Desktop GPU |
| Q5_K_M | ~3.0 GB | Desktop GPU |
| Q4_K_M | ~2.5 GB | Mid-range GPU / CPU |
| Q3_K_M | ~2.0 GB | Laptop GPU |
| Q2_K | ~1.5 GB | CPU-only |
Usage
llama.cpp
./main -m model_q4_k_m.gguf -n 512 -p "What does the Bible say about..."
llama-cpp-python
from llama_cpp import Llama
llm = Llama(model_path="model_q4_k_m.gguf", n_gpu_layers=-1)
response = llm("What does the Bible say about...", max_tokens=512)
print(response["choices"][0]["text"])
LM Studio
- Download any
.gguffile from this repo - Open LM Studio → Load Model
- Select and chat
Ollama
ollama pull hf.co/tinyopsec/bible-assistant-qwen35-4b-GGUF:q4_k_m
ollama run bible-assistant-qwen35-4b-GGUF:q4_k_m
Original Model
See Ttimms/Bible-Assistant-Qwen3.5-4B-v2 for model details, training data, and capabilities.
License
Licensed under Apache 2.0. See original model repo for details.