grug-35b-qat-q4-gguf
35b MoE brother in QAT four-bit rock. brain feel rounding rock during training so Q4 squish hurt less.
recipe: expert-freeze QAT on grug-35b-v2 - ALL text linear (expert include) fake-quant int4 g32 in forward, gradient only flow to attention/DeltaNet/shared path (1.4B trainable; expert too heavy for one cave GPU). ~1.7M grug token, Adafactor lr 2e-6. release = 25% QAT + 75% original anchor (grug family standard). fresh bf16 export, ONE Q4_K_M squish.
rocks
| file | what |
|---|---|
| grug-35b-qat-Q4_K_M.gguf | the QAT rock (~21 GB) |
| mmproj-grug-35b-v2-f16.gguf | eye rock (vision) |
if rock act broken
single-token spam = context-shift corruption, not rock. recent llama.cpp +
-c 16384+ for agent frontends. see main gguf card for full troubleshoot.
how run
llama-server -m grug-35b-qat-Q4_K_M.gguf --mmproj mmproj-grug-35b-v2-f16.gguf \
-c 16384 --temp 0.6 --top-p 0.95 --top-k 20
ordinary rocks: grug-35b-v2-gguf. 27b QAT brother: grug-27b-qat-q4-gguf. grug made by ProCreations.