adtcFixcontext-agri-sft-qwen3-0.6b-GGUF

GGUF-quantized versions of EYEDOL/adtc-agri-sft-qwen3-0.6b for use with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes.

Files

Pick Q4_K_M for the smallest footprint, Q6_K for the closest fidelity to the original fp16 model, or Q5_K_M as a middle ground.

⚠️ Runtime memory: read before deploying

This model uses grouped-query attention with 8 KV heads and 28 layers, so its KV cache is much larger relative to its size than typical small models. The model's advertised context is 40,960 tokens, and previous testing showed that launching with no explicit -c produced roughly 4 GB of RAM from an F16 KV cache alone — even though the Q4_K_M weight file is only ~378 MB.

This build's default context metadata has been reduced to 4096 tokens so that a launch which omits -c falls back to something reasonable instead of 40,960. This does not replace explicit runtime flags — -c, -ctk, -ctv, -b, -ub still need to be set for a tuned deployment; it only makes the forgotten-flag case safe.

Recommended low-RAM launch (llama.cpp):

./build/bin/llama-cli \
  -m model-Q4_K_M.gguf \
  -p "your prompt here" \
  -n 512 -c 4096 -ctk q8_0 -ctv q8_0 -b 512 -ub 128 -np 1

Q4 KV (-ctk q4_0 -ctv q4_0) was tested and produced a severe quality regression for this model — Q8_0 KV is the recommended floor for acceptable output.

Usage

Ollama

# Modelfile
FROM ./model-Q4_K_M.gguf
PARAMETER num_ctx 4096

ollama create adtc-agri-qwen3 -f Modelfile
ollama run adtc-agri-qwen3 "Your farming question here"

Python (llama-cpp-python)

from llama_cpp import Llama
llm = Llama(model_path="model-Q4_K_M.gguf", n_ctx=4096, type_k=8, type_v=8)
out = llm("Your farming question here", max_tokens=512)
print(out["choices"][0]["text"])

Intended use & limitations

An offline agricultural advisory baseline for smallholder farmers and extension officers in Nigeria. Not validated for general-purpose assistant use; outputs should be treated as advisory rather than authoritative agronomic or financial guidance — always recommend in-person expert follow-up for high-stakes decisions.

---

Converted with llama.cpp's convert_hf_to_gguf.py + llama-quantize, context metadata adjusted with gguf_set_metadata.py, generated 2026-09-22 07:25 UTC.