G9v3-3B GGUF

GGUF quantized versions of ai9stars/G9v3-3B, a compact 3B-parameter Llama-architecture language model supporting English and Chinese, with a 128K context window, tool-calling, and a built-in thinking/reasoning mode.

Model Overview

G9v3-3B is a lightweight text-generation model built on the Llama architecture. Despite its small size (~3B parameters), it supports a 131,072-token context window (128K) thanks to a high RoPE θ of 5,000,000. The model uses Grouped-Query Attention (GQA) with 2 key-value heads across 16 attention heads for efficient inference, and includes special tokens for thinking/reasoning (<|thought_begin|> / <|thought_end|>, /think, /no_think) and tool calling (<function>, <|tool_call|>, etc.).

The model uses a ChatML-style conversation format with <|im_start|> / <|im_end|> delimiters.

Model Architecture

Property	Value
Architecture	Llama
Parameters	~3B
Hidden Size	2048
Intermediate Size	6144
Layers	52
Attention Heads	16
KV Heads	2 (GQA)
Head Dimension	128
Context Length	131,072
Vocabulary Size	130,560
RoPE Theta	5,000,000
Original Precision	bfloat16
Supported Languages	en, zh

Available GGUF Files

File	Quantization	Size	Use Case
`g9v3-3b-bf16.gguf`	BF16	5.6 GB	Max precision, reference model
`g9v3-3b-Q8_0.gguf`	Q8_0	3.0 GB	Near-lossless, good speed/quality balance
`g9v3-3b-Q6_K.gguf`	Q6_K	2.3 GB	Very high quality, recommended for quality
`g9v3-3b-Q5_K_M.gguf`	Q5_K_M	2.0 GB	High quality, balanced
`g9v3-3b-Q5_K_S.gguf`	Q5_K_S	2.0 GB	High quality, slightly smaller than M
`g9v3-3b-Q4_K_M.gguf`	Q4_K_M	1.8 GB	Good quality, recommended default
`g9v3-3b-Q4_K_S.gguf`	Q4_K_S	1.7 GB	Smaller, acceptable quality
`g9v3-3b-Q4_0.gguf`	Q4_0	1.7 GB	Legacy quant, fastest inference

Recommended: Q4_K_M or Q5_K_M offer the best quality-to-size trade-off for most use cases.

Usage

llama.cpp CLI

./llama-cli \
  -m g9v3-3b-Q4_K_M.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 0.9 --top-p 0.95

llama-server (OpenAI-compatible API)

./llama-server \
  -m g9v3-3b-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080

Thinking Mode

The model supports a thinking/reasoning mode controlled via special tokens. Use /think to enable extended reasoning or /no_think to disable it. When thinking is enabled, the model outputs its reasoning between <|thought_begin|> and <|thought_end|> tokens before providing the final answer.

Tool Calling

The model supports function/tool calling via XML-style <function> tags. Tool definitions are injected into the system prompt, and the model responds with <function name="..."> blocks containing <param> elements.

Generation Parameters

Recommended parameters from the original model's generation_config.json:

Parameter	Value
Temperature	0.9
Top-P	0.95
BOS Token ID	0
EOS Token IDs	1, 130073
Pad Token ID	1

Quantization

These GGUF files were created from the BF16 source model using llama-quantize from the llama.cpp project.

Acknowledgements

Original model: ai9stars/G9v3-3B
Quantization tool: llama.cpp

License

Apache License 2.0