grug-27b-qat-q4-gguf
2026-07-23 refresh: rock now QAT-trained from grug-27b v2.1 weights (deep-think + agent-discipline update). bench table below measured on the v1-era rock vs v1 control - directional guide still; v2.1 rock inherit both the QAT recovery AND the v2.1 brain improvements.
grug put 27b brain in four-bit cave DURING training. brain feel rounding rock before final squish. this not normal quant. this QAT recovery rock, made for Q4 people.
recipe (same as grug-9b-qat, scaled): full-weight QAT on grug-27b, fake int4 asymmetric group-32 with straight-through gradient, ~3M token of grug-think data, Adafactor LR 2e-6, 249 step on one H200. release = 25% QAT move + 75% original anchor (full QAT overcorrect, 9b teach grug this). then fresh BF16 export, quantize ONE time to Q4_K_M.
rocks
| file | what |
|---|---|
| grug-27b-qat-Q4_K_M.gguf | the QAT rock, ~16.5 GB |
| mmproj-grug-27b-f16.gguf | eye rock (vision), same as main gguf repo |
number. same cave, same harness, same llama.cpp build
three rock fight: ordinary Q4 (control), full-QAT Q4, and this rock (25% QAT blend). full-QAT win MBPP big but BREAK tool hand (agent valid 96.6 -> 82.8). grug no ship broken hand. blend rock best overall:
| test | control Q4 | full-QAT Q4 | THIS ROCK |
|---|---|---|---|
| MBPP-60 pass % | 81.7 | 91.7 | 88.3 (+6.6) |
| GSM8K-60 % | 96.7 | 95.0 | 98.3 (+1.6) |
| agent tool-call valid % | 96.6 | 82.8 | 96.6 (same) |
| agent right-tool % | 93.1 | 79.3 | 89.7 (-3.4) |
| agent args schema-valid % | 100 | 100 | 100 |
| identity loop rate % | 3.3 | 0.0 | 1.7 (halved) |
| greedy longform loops | 1 | 0 | 0 |
QAT feel rounding rock during training -> Q4 squish hurt less. coding and math UP, loop sickness DOWN, tool hand intact. small right-tool dip is the honest trade. grug show all numbers, hide nothing.
if rock act broken
single-token spam ("/" forever etc) = NOT the rock. hybrid DeltaNet brain CANNOT survive llama.cpp context-shift: old builds shift on context overflow and corrupt the recurrent state into token spam. fix:
- use RECENT llama.cpp (qwen3_5 support; new builds refuse instead of shift)
- agent frontends (OpenCode etc): set
-c 16384or bigger - still broken? re-download rock (verify size) + check backend grug re-test rock after every report: loads clean, zero spam at proper config.
how run
llama-server -m grug-27b-qat-Q4_K_M.gguf --mmproj mmproj-grug-27b-f16.gguf \
-c 16384 --temp 0.6 --top-p 0.95 --top-k 20
need recent llama.cpp (qwen3_5 arch). grug think live in <think>, short on
purpose. tool call use XML format. main model card:
grug-27b.