KittyLM-4B GGUF (gemma3-4b)
Quantized GGUF weights for KittyLM/kittylm-gemma3-4b — the kitten-speak LoRA finetune of google/gemma-3-4b-it. Finetune details, eval results, and license notes live on the main repo card. Training data: KittyLM/kittylm-data.
Files
kittylm-gemma3-4b-q4_k_m.ggufkittylm-gemma3-4b-q5_k_s.ggufkittylm-gemma3-4b-q6_k.ggufkittylm-gemma3-4b-q8_0.gguf
Quant roadmap: Q4_K_M (recommended daily driver), Q5_K_S (small quality bump over Q4_K_M), Q6_K (near-lossless, large), Q8_0 (near-F16 fidelity, biggest). Q4_K_M ships today and is the recommended daily driver;
higher-bit quants trade size for fidelity (see scripts/quantize_gguf.py).
Usage (Ollama, direct — recommended)
No Modelfile needed. This repo ships Ollama template (gemma family), system
(kitten persona), and params files, so the HF integration picks them up automatically:
ollama run hf.co/KittyLM/kittylm-gemma3-4b-gguf:Q4_K_M
# other quants (when present): :Q8_0, :Q6_K, :Q5_K_S
Usage (Ollama, classic Modelfile)
A ready-made Modelfile lives in this repo (FROM points at the Q4_K_M file):
hf download KittyLM/kittylm-gemma3-4b-gguf --include "*.gguf" --local-dir ./gguf
hf download KittyLM/kittylm-gemma3-4b-gguf Modelfile --local-dir ./gguf
ollama create kittylm-gemma3-4b -f ./gguf/Modelfile
ollama run kittylm-gemma3-4b "Good night!"
Usage (llama.cpp / LM Studio)
Point llama.cpp (llama-cli -m <file>.gguf) or LM Studio at the .gguf file directly.
Apply the model's chat template and keep the KittyLM system prompt (also shipped as the
system file in this repo and as system_prompt.txt in KittyLM/kittylm-data).