GGUF quantizations of Nanbeige4.2-3B for use with llama.cpp, KoboldCpp, LM Studio, Jan, Open WebUI, Ollama (via GGUF import), and other GGUF-compatible inference engines.
Base Model: Nanbeige/Nanbeige4.2-3B
This repository provides a collection of GGUF quantizations of Nanbeige4.2-3B optimized for local inference across a wide range of hardware configurations.
The model was converted from the original Hugging Face weights to GGUF format using the latest available llama.cpp conversion tools. Both traditional K-quants and importance-aware IQ-quants are included to provide a balance between quality, memory usage, and inference speed.
| Quant | Description |
|---|---|
| Q2_K | Fastest traditional 2-bit quant |
| Q2_K_S | Improved 2-bit K quant |
| IQ2_XXS | Ultra-small IQ quant |
| IQ2_XS | Small IQ quant |
| IQ2_S | Balanced IQ quant |
| IQ2_M | Highest-quality 2-bit IQ quant |
| Quant | Description |
|---|---|
| IQ3_XXS | Compact 3-bit IQ quant |
| IQ3_XS | Improved 3-bit IQ quant |
| IQ3_S | Balanced 3-bit IQ quant |
| IQ3_M | Highest-quality 3-bit IQ quant |
| Q3_K_S | Small K quant |
| Q3_K_M | Recommended quality/size balance |
| Q3_K_L | Highest-quality 3-bit K quant |
| Quant | Description |
|---|---|
| IQ4_XS | Efficient 4-bit IQ quant |
| IQ4_NL | High-quality nonlinear 4-bit quant |
| Q4_0 | Legacy 4-bit quant |
| Q4_1 | Improved legacy 4-bit quant |
| Q4_K_S | Small K quant |
| Q4_K_M | Recommended 4-bit quant |
| Quant | Description |
|---|---|
| Q5_K_S | Small 5-bit K quant |
| Q5_K_M | Recommended 5-bit K quant |
| Quant | Description |
|---|---|
| Q6_K | High-quality 6-bit quant |
| Quant | Description |
|---|---|
| Q8_0 | Near-lossless quantization |
| Quant | Description |
|---|---|
| BF16 | Original BF16 GGUF conversion |
./llama-cli \
-m Nanbeige4.2-3B-Q4_K_M.gguf \
-p "Explain quantum computing in simple terms."
Create a Modelfile:
FROM ./Nanbeige4.2-3B-Q4_K_M.gguf
Then:
ollama create nanbeige4-3b -f Modelfile
Run:
ollama run nanbeige4-3b
llama-quantize.llama.cpp release at build time.All model weights, architecture, training, and tokenizer credits belong to the original authors of:
Nanbeige/Nanbeige4.2-3B
Converted and quantized for the community by:
Nithin Sai Kumar (NANI-Nithin)
This repository only provides converted and quantized GGUF files. Please refer to the original model repository for:
Users must comply with the original model license and usage restrictions.
If you find this model useful, please consider supporting the original creators by starring, downloading, and contributing feedback to the original repository:
有文件?提交种子。
提交种子Nanbeige/Nanbeige4.2-3B