Piccolo-9B-Q4_K_M
Piccolo-9B-Q4_K_M is the Q4_K_M GGUF release of Piccolo-9B for local tool review and context selection.
Download
hf download agentscope-ai/Piccolo-9B-Q4_K_M --local-dir Piccolo-9B-Q4_K_M
Precision and conversion
The text backbone was converted to BF16 GGUF, then quantized to Q4_K_M. Download all 2 shards together and load Piccolo-9B-Q4_K_M-00001-of-00002.gguf.
Required decision inference
Use the Piccolo-compatible llama.cpp runtime with bidirectional prompt attention and native candidate scoring. The /v1/systemone endpoint returns probabilities without generating answer tokens.
Numerical validation
Across 556 synthetic judgments, mean total variation from the BF16 reference was 0.013701 (p95 0.081681). Validation details document the comparison.
Evaluation
Piccolo-9B benchmark results · All models
This model is also available on ModelScope.