中文Sign up

WhiskyAKM/G9v3-3B-GGUF

🤗 Hugging Face sourcetext-generationapache-2.03B activated21 GBGGUF✓ 8 checksumsupdated today
Submit in one command

Run it next to your model folder. It makes the torrent, checks your files against Hugging Face, and submits it. You just start seeding and paste your key from your account. It only reads your files and never changes them. Read the script first if you like.

curl -fsSL https://pirateface.co/package.sh | bash -s -- --repo WhiskyAKM/G9v3-3B-GGUF ./model-folder
Needs a seeder →

G9v3-3B GGUF

GGUF quantized versions of ai9stars/G9v3-3B, a compact 3B-parameter Llama-architecture language model supporting English and Chinese, with a 128K context window, tool-calling, and a built-in thinking/reasoning mode.

Model Overview

G9v3-3B is a lightweight text-generation model built on the Llama architecture. Despite its small size (~3B parameters), it supports a 131,072-token context window (128K) thanks to a high RoPE θ of 5,000,000. The model uses Grouped-Query Attention (GQA) with 2 key-value heads across 16 attention heads for efficient inference, and includes special tokens for thinking/reasoning (<|thought_begin|> / <|thought_end|>, /think, /no_think) and tool calling (<function>, <|tool_call|>, etc.).

The model uses a ChatML-style conversation format with <|im_start|> / <|im_end|> delimiters.

Model Architecture

Property Value
Architecture Llama
Parameters ~3B
Hidden Size 2048
Intermediate Size 6144
Layers 52
Attention Heads 16
KV Heads 2 (GQA)
Head Dimension 128
Context Length 131,072
Vocabulary Size 130,560
RoPE Theta 5,000,000
Original Precision bfloat16
Supported Languages en, zh

Available GGUF Files

File Quantization Size Use Case
g9v3-3b-bf16.gguf BF16 5.6 GB Max precision, reference model
g9v3-3b-Q8_0.gguf Q8_0 3.0 GB Near-lossless, good speed/quality balance
g9v3-3b-Q6_K.gguf Q6_K 2.3 GB Very high quality, recommended for quality
g9v3-3b-Q5_K_M.gguf Q5_K_M 2.0 GB High quality, balanced
g9v3-3b-Q5_K_S.gguf Q5_K_S 2.0 GB High quality, slightly smaller than M
g9v3-3b-Q4_K_M.gguf Q4_K_M 1.8 GB Good quality, recommended default
g9v3-3b-Q4_K_S.gguf Q4_K_S 1.7 GB Smaller, acceptable quality
g9v3-3b-Q4_0.gguf Q4_0 1.7 GB Legacy quant, fastest inference

Recommended: Q4_K_M or Q5_K_M offer the best quality-to-size trade-off for most use cases.

Usage

llama.cpp CLI

./llama-cli \
  -m g9v3-3b-Q4_K_M.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 0.9 --top-p 0.95

llama-server (OpenAI-compatible API)

./llama-server \
  -m g9v3-3b-Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080

Thinking Mode

The model supports a thinking/reasoning mode controlled via special tokens. Use /think to enable extended reasoning or /no_think to disable it. When thinking is enabled, the model outputs its reasoning between <|thought_begin|> and <|thought_end|> tokens before providing the final answer.

Tool Calling

The model supports function/tool calling via XML-style <function> tags. Tool definitions are injected into the system prompt, and the model responds with <function name="..."> blocks containing <param> elements.

Generation Parameters

Recommended parameters from the original model's generation_config.json:

Parameter Value
Temperature 0.9
Top-P 0.95
BOS Token ID 0
EOS Token IDs 1, 130073
Pad Token ID 1

Quantization

These GGUF files were created from the BF16 source model using llama-quantize from the llama.cpp project.

Acknowledgements

License

Apache License 2.0