adtcFixcontext-agri-sft-qwen3-0.6b-GGUF
GGUF-quantized versions of EYEDOL/adtc-agri-sft-qwen3-0.6b for use
with llama.cpp, Ollama,
LM Studio, and other GGUF-compatible runtimes.
Files
| Quantization | File | Size | Notes |
|---|---|---|---|
| Q4_K_M | model-Q4_K_M.gguf |
0.40 GB | 4-bit K-quant, medium — smallest file, good default for CPU/edge inference. |
| Q5_K_M | model-Q5_K_M.gguf |
0.44 GB | 5-bit K-quant, medium — noticeably better quality than Q4_K_M for a modest size increase. |
| Q6_K | model-Q6_K.gguf |
0.50 GB | 6-bit K-quant — closest to fp16 quality of the three, largest file; best if you have the RAM/storage to spare. |
Pick Q4_K_M for the smallest footprint, Q6_K for the closest fidelity to the original fp16 model, or Q5_K_M as a middle ground.
⚠️ Runtime memory: read before deploying
This model uses grouped-query attention with 8 KV heads and 28 layers, so its
KV cache is much larger relative to its size than typical small models. The
model's advertised context is 40,960 tokens, and previous testing showed
that launching with no explicit -c produced roughly 4 GB of RAM from an
F16 KV cache alone — even though the Q4_K_M weight file is only ~378 MB.
This build's default context metadata has been reduced to 4096 tokens
so that a launch which omits -c falls back to something reasonable instead of
40,960. This does not replace explicit runtime flags — -c, -ctk, -ctv,
-b, -ub still need to be set for a tuned deployment; it only makes the
forgotten-flag case safe.
Recommended low-RAM launch (llama.cpp):
./build/bin/llama-cli \
-m model-Q4_K_M.gguf \
-p "your prompt here" \
-n 512 -c 4096 -ctk q8_0 -ctv q8_0 -b 512 -ub 128 -np 1
Q4 KV (-ctk q4_0 -ctv q4_0) was tested and produced a severe quality regression
for this model — Q8_0 KV is the recommended floor for acceptable output.
Usage
Ollama
# Modelfile
FROM ./model-Q4_K_M.gguf
PARAMETER num_ctx 4096
ollama create adtc-agri-qwen3 -f Modelfile
ollama run adtc-agri-qwen3 "Your farming question here"
Python (llama-cpp-python)
from llama_cpp import Llama
llm = Llama(model_path="model-Q4_K_M.gguf", n_ctx=4096, type_k=8, type_v=8)
out = llm("Your farming question here", max_tokens=512)
print(out["choices"][0]["text"])
Intended use & limitations
An offline agricultural advisory baseline for smallholder farmers and extension officers in Nigeria. Not validated for general-purpose assistant use; outputs should be treated as advisory rather than authoritative agronomic or financial guidance — always recommend in-person expert follow-up for high-stakes decisions.
Converted with llama.cpp's convert_hf_to_gguf.py + llama-quantize, context
metadata adjusted with gguf_set_metadata.py, generated
2026-09-22 07:25 UTC.