Nanbeige4.2-3B-GGUF
GGUF quantizations of Nanbeige4.2-3B for use with llama.cpp, KoboldCpp, LM Studio, Jan, Open WebUI, Ollama (via GGUF import), and other GGUF-compatible inference engines.
Base Model: Nanbeige/Nanbeige4.2-3B
About
This repository provides a collection of GGUF quantizations of Nanbeige4.2-3B optimized for local inference across a wide range of hardware configurations.
The model was converted from the original Hugging Face weights to GGUF format using the latest available llama.cpp conversion tools. Both traditional K-quants and importance-aware IQ-quants are included to provide a balance between quality, memory usage, and inference speed.
Available Quantizations
2-bit
| Quant | Description |
|---|---|
| Q2_K | Fastest traditional 2-bit quant |
| Q2_K_S | Improved 2-bit K quant |
| IQ2_XXS | Ultra-small IQ quant |
| IQ2_XS | Small IQ quant |
| IQ2_S | Balanced IQ quant |
| IQ2_M | Highest-quality 2-bit IQ quant |
3-bit
| Quant | Description |
|---|---|
| IQ3_XXS | Compact 3-bit IQ quant |
| IQ3_XS | Improved 3-bit IQ quant |
| IQ3_S | Balanced 3-bit IQ quant |
| IQ3_M | Highest-quality 3-bit IQ quant |
| Q3_K_S | Small K quant |
| Q3_K_M | Recommended quality/size balance |
| Q3_K_L | Highest-quality 3-bit K quant |
4-bit
| Quant | Description |
|---|---|
| IQ4_XS | Efficient 4-bit IQ quant |
| IQ4_NL | High-quality nonlinear 4-bit quant |
| Q4_0 | Legacy 4-bit quant |
| Q4_1 | Improved legacy 4-bit quant |
| Q4_K_S | Small K quant |
| Q4_K_M | Recommended 4-bit quant |
5-bit
| Quant | Description |
|---|---|
| Q5_K_S | Small 5-bit K quant |
| Q5_K_M | Recommended 5-bit K quant |
6-bit
| Quant | Description |
|---|---|
| Q6_K | High-quality 6-bit quant |
8-bit
| Quant | Description |
|---|---|
| Q8_0 | Near-lossless quantization |
Full Precision
| Quant | Description |
|---|---|
| BF16 | Original BF16 GGUF conversion |
Recommended Quantizations
For 4 GB RAM
- IQ2_M
- Q2_K_S
For 6 GB RAM
- IQ3_M
- Q3_K_M
For 8 GB RAM
- IQ4_NL
- Q4_K_M
For Maximum Quality
- Q6_K
- Q8_0
- BF16
Usage with llama.cpp
./llama-cli \
-m Nanbeige4.2-3B-Q4_K_M.gguf \
-p "Explain quantum computing in simple terms."
LM Studio
- Download your preferred GGUF file.
- Place it in the LM Studio models directory.
- Refresh models.
- Load and start chatting.
Ollama
Create a Modelfile:
FROM ./Nanbeige4.2-3B-Q4_K_M.gguf
Then:
ollama create nanbeige4-3b -f Modelfile
Run:
ollama run nanbeige4-3b
Quantization Notes
- K-Quants were generated using
llama-quantize. - IQ-Quants were generated using importance matrix (imatrix) quantization for improved quality retention at lower bitrates.
- Quantizations were produced using the latest available
llama.cpprelease at build time.
Credits
Base Model
All model weights, architecture, training, and tokenizer credits belong to the original authors of:
Nanbeige/Nanbeige4.2-3B
GGUF Conversion & Quantization
Converted and quantized for the community by:
Nithin Sai Kumar (NANI-Nithin)
Disclaimer
This repository only provides converted and quantized GGUF files. Please refer to the original model repository for:
- Training details
- Evaluation results
- License information
- Intended use guidance
- Safety considerations
Users must comply with the original model license and usage restrictions.
Support the Original Authors
If you find this model useful, please consider supporting the original creators by starring, downloading, and contributing feedback to the original repository: