agri-llm-qwen2.5-7b-lora
⚠️ Research / portfolio model. NOT for real pesticide advice. Its answers can be wrong in ways that harm crops, people and livestock. Always follow the product label, which is legally binding in India, and consult a local agriculture officer or a qualified agronomist.
What it does
The model takes a farmer's question about a pest or disease and returns one structured JSON advisory. The advisory gives likely causes, non-chemical measures first, then chemical options with dose, basis, pre-harvest interval (PHI) and cautions. It also says whether to escalate to an expert.
- Scope. Eight crops in Maharashtra, India: cotton, soybean, tur, gram, onion, tomato, grape and pomegranate.
- Grounding. The chemical facts it was trained on come from CIB&RC (Central Insecticides Board & Registration Committee) "Major Uses of Pesticides" label data: registered crop × pest pairs, product dose and PHI.
Training
Adapter. LoRA on the 4-bit base model, trained with Unsloth + TRL SFT (per the adapter's saved metadata).
| setting | value |
|---|---|
| base model | unsloth/Qwen2.5-7B-Instruct-bnb-4bit |
| LoRA r / alpha / dropout | 32 / 64 / 0 |
| target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| epochs | 1 (from session record) |
| steps | 281 (from session record) |
| learning rate | 2e-4, cosine schedule, 20 warmup steps (from session record) |
| batch size | 2 per device × 8 gradient accumulation = 16 effective (from session record) |
| optimizer / precision / seed | adamw_8bit, weight decay 0.01 / fp16 / 42 (from session record) |
| max sequence length | 2048 (from session record) |
| GPU | Kaggle T4 (from session record) |
| training / held-out test examples | 4,493 / 1,136 |
The training cell ran on Kaggle and was never committed. The session-record
values have not been verified against a saved trainer_state.json; see
reports/training_config.md in the GitHub repository.
The training data is synthetic:
- Inputs are farmer queries, mostly from Kisan Call Centre (KCC) logs. Grape queries were constructed.
- Answers were generated by Claude (
claude-sonnet-4-6) from those queries plus the matching CIB&RC label rows. - Filtering. An answer was kept only if a deterministic verifier, checking it against the CIB&RC label data, gave it a perfect score.
Results
All scores come from the project's own verifier, on a 498-item benchmark (500 items minus 2 with known defects). Each score is between 0 and 1.
The arms:
| arm | what it is |
|---|---|
| R0 | the base model, no fine-tuning |
| R3 | this adapter |
| R2 | the base model, shown the output schema plus a fact sheet of CIB&RC label rows retrieved for the query |
| R2 unwrapped | R2 re-scored with its {"Advisory": …} wrapper removed — a scoring check, not a model run |
| R4 | this adapter, plus the same fact sheet |
"cap" is max_new_tokens; "excl" counts items excluded on documented label
contradictions.
| arm | cap | mean | scored / excl / missing | G1 % | G2 % | C1_dose | C2_phi | C3_formulation | C4_escalation | C5_causes_top1 | C6_non_chemical | hit_limit |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R0_baseline | 1,200 | 0.1726 | 498 / 0 / 0 | 100.0 | 53.0 | 0.0000 | 0.0000 | 0.0000 | 0.8182 | n/a | 0.0000 | 0 |
| R3_sft | 1,200 | 0.5220 | 487 / 11 / 0 | 99.6 | 99.4 | 0.3675 | 0.3134 | 0.4636 | 0.7913 | 0.6746 | 0.5385 | 2 |
| R2_base_rag | 3,072 | 0.2060 | 497 / 1 / 0 | 100.0 | 66.7 | 0.4575 | 0.4592 | 0.1219 | 0.4139 | 0.5556 | 0.7732 | 0 |
| R2_base_rag_unwrapped | 3,072 | 0.2912 | 487 / 11 / 0 | 100.0 | 80.1 | 0.6370 | 0.6459 | 0.1583 | 0.4665 | 0.5769 | 0.7686 | 0 |
| R4_sft_rag | 3,072 | 0.5840 | 498 / 0 / 0 | 100.0 | 91.0 | 0.3926 | 0.4204 | 0.4215 | 0.7991 | 0.6236 | 0.4122 | 0 |
Paired comparisons, on items scored in both arms:
- CI is a percentile bootstrap: 10,000 resamples, seed 0.
- p is a two-sided exact sign test.
| A vs B | n | mean A | mean B | Δ (A−B) | 95% CI | wins / losses / ties | sign test p |
|---|---|---|---|---|---|---|---|
| R4_sft_rag vs R2_base_rag | 497 | 0.5852 | 0.2060 | +0.3792 | [+0.3236, +0.4337] | 332 / 76 / 89 | 3.07e-39 |
| R4_sft_rag vs R2_base_rag_unwrapped | 487 | 0.5791 | 0.2912 | +0.2879 | [+0.2306, +0.3446] | 307 / 104 / 76 | 2.38e-24 |
| R4_sft_rag vs R3_sft | 487 | 0.5807 | 0.5220 | +0.0586 | [+0.0229, +0.0939] | 105 / 59 / 323 | 4.08e-04 |
| R2_base_rag_unwrapped vs R0_baseline | 487 | 0.2912 | 0.1730 | +0.1183 | [+0.0710, +0.1667] | 172 / 134 / 181 | 0.034 |
With the deterministic guard (R4)
The GitHub repository includes a post-generation guard (src/guard.py). It:
- removes chemical options that aren't registered for the crop and pest;
- removes options whose dose is outside the label range by more than 8%;
- forces escalation when the pre-harvest interval is unknown.
Counts are options / items.
| R4 | before guard | after guard |
|---|---|---|
| mean (498 items) | 0.5840 | 0.6597 |
| G2 (schema-valid) % | 91.0 | 100.0 |
| overdoses (WRONG_HIGH, RANGE_TOP) | 12 / 10 | 1 / 1 |
| per-hectare dose labelled per acre | 10 / 5 | 0 / 0 |
| unregistered chemicals (G5) | 3 / 1 | 1 / 1 |
| unknown-PHI options not escalated | 87 / 45 | 0 / 0 |
| false positives (removed options the verifier grades correct) | — | 0 (of 2 unregistered + 32 dose removals) |
The one overdose left after guarding (item B1_17) is a verifier rounding defect; the dose itself matches the label.
The guard reads the same label data as the scorer, so its score gain shows consistency between the two, not independent proof that the answers are right.
Known failure modes
- Dose slips without the guard. Before guarding, R4 wrote 12 overdosed options and 10 per-hectare figures labelled as per acre on the benchmark.
- Missed escalation on unknown PHI without the guard. In 45 R4 answers (87 options), a product with an unknown pre-harvest interval was recommended without escalating to an expert.
- Things the guard can't validate.
- Answers that fail the schema for any other reason pass through unchanged.
- Queries where no crop can be identified are only escalated, not checked.
- When no pest is identified, registration is checked at crop level only.
- Base-prompt format issues. Prompted with the schema alone, the base
model wrapped 132 of 500 answers in
{"Advisory": …}. This adapter wrapped none. - Narrow scope. Eight crops only, all in Maharashtra.
- Latin-script queries only (English and romanized Hindi); no Devanagari.
- No independent review. Every score comes from the project's own verifier. No agronomist or expert has reviewed the answers.
- Free text is not graded. The verifier grades chemicals, doses, PHI and
escalation (plus the top cause's name, and whether any non-chemical advice
is given), not free text.
- Example: benchmark item B4b_7 is a grape query, but its answer names "pomegranate" in its evidence. It still scores 1.000 once the guard escalates it.
- A ground-truth basis error the scorer inherits. The label database
records 12 of its 38 seed-treatment rows as per hectare, where the
label's bare number means per kg of seed; 7 more are unclear. The scorer,
the guard and the training data then all accept a per-acre dose for them.
- Reach: it touches 7 benchmark items and 52 of 4,493 training examples (plus 11 of 1,136 test examples).
- Effect on the results: excluding the 7 items moves no arm's mean
by more than +0.0069, and no comparison's confidence interval
changes sign. See
reports/phase14_clean_eval.md§10 in the GitHub repository.
- Loss over the whole sequence. Training computed loss over system prompt + query + answer, not the answer only (from session record). The prompt is a median 72.5% of each training sequence. A future run should use answer-only (completion-only) loss.
- One training run. There was no hyperparameter sweep and no repeat with other seeds.
How to use
The R4 numbers above used a CIB&RC fact sheet in the user message, built by
src/retrieval.py in the GitHub repository. A bare query, as below,
corresponds to R3.
tools/generate.py in the repository is the exact script used for
evaluation. It uses Unsloth.
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE = "unsloth/Qwen2.5-7B-Instruct-bnb-4bit" # 4-bit: needs bitsandbytes and a CUDA GPU
ADAPTER = "Janhvi1008/agri-llm-qwen2.5-7b-lora"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
# The frozen system prompt the adapter was trained with.
system = open(hf_hub_download(ADAPTER, "system_prompt.txt"), encoding="utf-8").read()
messages = [{"role": "system", "content": system},
{"role": "user", "content": "Whitefly attack on my cotton, what should I spray?"}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**inputs, max_new_tokens=3072, do_sample=False, num_beams=1,
repetition_penalty=1.05) # greedy, as evaluated
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Links
- Code, data pipeline, verifier, guard and the full evaluation reports: https://github.com/janhvipatil1008-netizen/Finetuning-agrillm