G9v3-3B-GGUF
GGUF quantized releases of ai9stars/G9v3-3B for llama.cpp and compatible runtimes.
Model Information
- Base Model: ai9stars/G9v3-3B
- Parameter Size: 3B
- Format: GGUF
- Quantized By: NANI-Nithin
- Quantization Tool: llama.cpp
Available Files
2-bit
- Q2_K
- IQ2_M
- Q2_K_L
3-bit
- IQ3_XXS
- IQ3_XS
- Q3_K_S
- IQ3_M
- Q3_K_M
- Q3_K_L
- Q3_K_XL
4-bit
- IQ4_XS
- IQ4_NL
- Q4_0
- Q4_1
- Q4_K_S
- Q4_K_M
5-bit
- Q5_K_S
- Q5_K_M
6-bit
- Q6_K
- Q6_K_L
8-bit
- Q8_0
Full Precision
- F16/BF16 GGUF
Recommended Quantizations
Best Overall
Q4_K_M
Recommended for most users. Excellent balance of quality, memory usage, and speed.
Higher Quality
Q5_K_M or Q6_K
For users seeking maximum quality while still benefiting from quantization.
Best IQ Quant
IQ4_NL
Excellent quality-per-GB and one of the strongest modern importance-aware quantizations.
Low Memory Systems
IQ3_M or Q3_K_M
Good balance of usability and reduced memory requirements.
IQ Quantizations
The following importance-aware quantizations are included:
- IQ2_M
- IQ3_XXS
- IQ3_XS
- IQ3_M
- IQ4_XS
- IQ4_NL
These quantizations were generated using an importance matrix (imatrix) calibration pass and typically provide improved quality retention compared to traditional quantization methods at similar file sizes.
Usage
llama.cpp
./llama-cli \
-m G9v3-3B-Q4_K_M.gguf \
-p "Hello!"
Ollama
Create a Modelfile:
FROM ./G9v3-3B-Q4_K_M.gguf
Then:
ollama create g9v3-3b -f Modelfile
ollama run g9v3-3b
LM Studio
Download the desired GGUF file and import it directly into LM Studio.
Credits
- Original Model: ai9stars/G9v3-3B
- GGUF Conversion & Quantization: NANI-Nithin
- Quantization Framework: llama.cpp
Disclaimer
This repository contains converted GGUF files only.
Please refer to the original model repository for licensing terms, training methodology, benchmark results, intended use, limitations, and safety information.
Original model: