grug-9b-gguf
grug squish grug-9b into GGUF rocks. small rock fit small cave. all rock think 11-word grug thinks. all rock code.
grug-9b = Ornith-1.0-9B taught to reason SHORT: think tokens -94%, whole benchmark 3.3x faster, MBPP held (80->78), tool-picking better (+11). full story + honest tradeoffs on grug-9b card.
which rock for which cave
| rock | size (~) | quality | cave |
|---|---|---|---|
| Q8_0 | ~9.7 GB | basically lossless | 16GB+ VRAM or Mac |
| Q6_K | ~7.5 GB | very close | 12GB VRAM |
| Q5_K_M | ~6.5 GB | good | 10GB VRAM |
| Q4_K_M | ~5.5 GB | good, most popular rock | 8GB VRAM (RTX 4060 cave!) |
| Q3_K_M | ~4.4 GB | okay, some brain lost | small cave, phone-adjacent |
grug advice: Q4_K_M default. Q8_0 if cave big. below Q3, bird forget how code — grug not ship those.
how run
llama.cpp (need RECENT build — qwen3_5 architecture very new, old build no understand bird):
llama-cli -hf ProCreations/grug-9b-gguf:Q4_K_M
LM Studio: search grug-9b-gguf, pick rock, go.
warnings from grug
- text only. vision tower not come into GGUF cave. use full grug-9b weights if need eyes
- need llama.cpp from after qwen3_5 support land. if error say unknown architecture: update
- thinking come in
<think>tag, short on purpose. that is whole point. not bug. grug proud