Phi-3 Agricultural Analyst (LoRA experiment)

Personal experiment (December 2025). This was my first hands-on LoRA project: a small adapter that teaches Phi-3-mini a structured report format for agricultural questions about 8 US counties. It is a learning artifact, not a production model. Please read What I got wrong before using it.

Summary


Base model	`microsoft/Phi-3-mini-4k-instruct` (3.8B)
Method	LoRA SFT (TRL `SFTTrainer`), BF16
Data	1,000 synthetic Q&A examples (900 train / 100 validation), 8 US counties
Hardware / time	NVIDIA DGX Spark (GB10), ~51 minutes
Adapter actually trained	`o_proj` + `down_proj` only, 35,651,584 parameters (≈0.93% of the base), 142.6 MB (FP32 safetensors)

What I got wrong (lessons learned)

Only 2 of the 7 listed modules were actually adapted. adapter_config.json lists q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. But Phi-3 uses fused projections (qkv_proj and gate_up_proj), so PEFT matched only o_proj and down_proj. The adapter weights confirm it: 64 tensors for each of those two modules and nothing else (32 layers × 2 modules × LoRA A/B). The correct targets for Phi-3 would have been:
```
target_modules = ["qkv_proj", "o_proj", "gate_up_proj", "down_proj"]   # or "all-linear"
```
What I do now: call model.print_trainable_parameters() and list the LoRA-wrapped module names before every run.
The near-zero loss means the model memorized the synthetic template, not that it is very good. Final eval loss was 0.0186 and token accuracy 99.3%. The data was synthetic and highly templated (training loss fell from 1.605 to 0.115 within the first epoch). The 100 validation examples come from the same generator, so the low loss mostly shows the model learned that template.
The task evaluation is small and I didn't document it well. "Content Coverage" and "Structure Quality" were scored with my own rubric against the base model. I didn't record the number of items or the exact scoring procedure.
The response-time difference isn't a speedup. The fine-tuned model's responses took less time end to end (62.4 s vs 101.9 s). A LoRA adapter doesn't make each token faster. The likely reason is that the fine-tuned model writes shorter, more structured answers. I didn't measure tokens/s or output length separately.

Results (base vs fine-tuned, my rubric)

Metric	Base Phi-3-mini	Fine-tuned	Change
Content Coverage	50.0%	63.3%	+13.3 pp (+26.7% relative)
Structure Quality	56.7%	83.3%	+26.6 pp (+47.1% relative)
Avg end-to-end response time	101.9 s	62.4 s	−38.8% (see note 4 above)

Training / eval loss by epoch (training loss start → end of epoch):

Epoch	Training loss
1	1.605 → 0.115
2	0.115 → 0.020
3	0.020 → 0.019
Final eval loss	0.0186 (token accuracy 99.3%)

Training configuration (from `training_args.bin` and `adapter_config.json`)

Parameter	Value
LoRA rank / alpha / dropout	64 / 128 / 0.05
Modules effectively trained	`o_proj`, `down_proj` (see above)
Epochs	3
Learning rate / schedule	2e-4, linear, warmup ratio 0.1
Per-device batch × grad accumulation	16 × 2 = 32 effective
Optimizer	`adamw_torch_fused`, weight decay 0
Max sequence length	2048, no packing
Loss	computed on the full sequence (`assistant_only_loss=False`)
Precision	BF16, gradient checkpointing on
Eval / save	every epoch, best checkpoint loaded at end
Seed	42
Library versions	PEFT 0.18.0

Counties covered: Fresno CA, Kern CA, Lancaster PA, Sioux IA, Yakima WA, Imperial CA, Monterey CA, Deaf Smith TX.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

model_name = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, device_map="auto",
    trust_remote_code=True, attn_implementation="eager",
)
model = PeftModel.from_pretrained(base_model, "sarathi-balakrishnan/phi3-agricultural-analyst")

prompt = """You are an agricultural analyst. Analyze the following query using the provided data context.

### Query
What are the key investment opportunities in Fresno County agriculture?

### Data Context
County: Fresno, CA
Operators: 5,847
Average Farm Size: 423 acres
Irrigation Coverage: 89%
Revenue per Acre: $2,340

### Analysis"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=500, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

(The data-context numbers above are illustrative example inputs, not verified statistics.)

To merge for deployment: model = model.merge_and_unload().

Output format

The model was trained to produce: a regional summary, key insights, opportunities, risk factors, and a confidence level.

Limitations

Trained on synthetic data for 8 counties only; expect poor behaviour outside that template.
Only o_proj / down_proj were adapted, so this is a much weaker adapter than the config implies.
The evaluation is a small, self-scored rubric. There is no held-out set of real questions.
Any facts or figures in the output are not verified. This is not agricultural, financial, or investment advice.

What I'd do next

Retrain with the correct fused-module targets, use loss on assistant tokens only, run a rank sweep (r = 8 / 16 / 64), and evaluate on real, human-written questions with a documented rubric.

License

MIT (same as the Phi-3-mini base model).

sarathi-balakrishnan/phi3-agricultural-analyst