grug-9b-qat-q4
grug put 9b bird in four-bit cave during training. bird feel rounding rock before final squish. then grug make real Q4_K_M GGUF. this not normal squish. this QAT recovery.
QAT start from full grug-9b, never train old GGUF, never requantize quant rock. grug fake-quant text linear weight as asymmetric int4, group 32, straight-through gradient. train 3,628,102 token from grug-think. full QAT overcorrected, so grug keep 25% QAT move and 75% original anchor. fixed eval gate choose before release. then export fresh BF16 GGUF and quantize one time to Q4_K_M.
numbers. same cave, same harness
both rocks run same llama.cpp build, greedy, same prompt. control = ordinary grug-9b-Q4_K_M. QAT = this rock.
| test | old Q4 | QAT Q4 | change |
|---|---|---|---|
| HumanEval pass@1 % | 75.6 | 78.0 | +2.4 |
| MBPP pass@1 % | 75.0 | 74.0 | -1.0 |
| agent tool-call valid % | 61.1 | 88.9 | +27.8 |
| agent RIGHT tool % | 55.6 | 77.8 | +22.2 |
| agent reasoning present % | 100.0 | 100.0 | +0.0 |
| agent reasoning mean word | 7.1 | 5.8 | -1.3 |
grug reasoning format stay same: short grug brain inside <think>...</think>, normal say-word outside, XML tool call untouched. grug measure presence and length above, not merely hope.
grug honest: QAT help model adapt to 4-bit rounding. QAT not magic promise. numbers above say what happen on these probe. hardest real repo task may act different.
rock
grug-9b-qat-Q4_K_M.gguf — about 5.6 GB. good default for 8 GB VRAM cave.
llama-cli -hf ProCreations/grug-9b-qat-q4:Q4_K_M
need recent llama.cpp with qwen3_5 support. thinking live in <think> tag and stay short on purpose. tool call use model XML format.
recipe. grug show work
- source:
ProCreations/grug-9b - data:
ProCreations/grug-think - QAT: full text-stack linear weights, fake int4 asymmetric group size 32
- training: 684 optimizer step, 3,628,102 token, LR 2e-06
- release blend: 25% QAT checkpoint + 75% original checkpoint. grug show this because honest rock better than secret rock
- target export: llama.cpp Q4_K_M, quantized from fresh BF16 conversion
- vision tower not in GGUF. text bird only
training config and measured JSON live in this repo. grug no hide recipe behind bush.