🌿 Gemma 4 E4B-IT — Multimodal Plant Pathology & Spatial Grounding (rsLoRA)
This repository contains the fine-tuned Multimodal Rank-Stabilized LoRA (rsLoRA) weights for unsloth/gemma-4-E4B-it, trained on PandyaJeet/gemma4-plant-pathology-3k to perform:
- Macroscopic Visual Symptom Chain-of-Thought (
visual_evidence) - Spatial Lesion Bounding Box Localization (
[ymin, xmin, ymax, xmax]on a0–1000scale) - Differential Pathology (
ruled_out) & Canonical Diagnosis (15locked classes across Tomato, Potato, Corn, and OOD Guardrails)
📊 Training Hyperparameters
- Base Model:
unsloth/gemma-4-E4B-it(4-bit NF4 base +float32rsLoRA adapters) - Adapter Target Modules: Vision Encoder + Multimodal Projector + Language Decoder (
39,481,344trainable parameters /0.4947%) - LoRA Config:
r = 16,lora_alpha = 32,use_rslora = True,lora_dropout = 0.0 - Training Dataset: 3,375 stratified multimodal samples (225 per class across 15 classes)
- Final Training Loss:
~0.090 - 0.139(Assistant-only response loss masking)
🚀 Quickstart Inference (Unsloth)
import json
from PIL import Image
from unsloth import FastVisionModel
model, tokenizer = FastVisionModel.from_pretrained(
model_name="PandyaJeet/gemma4-e4b-plant-pathology-lora",
load_in_4bit=True,
)
FastVisionModel.for_inference(model)
SYSTEM_PROMPT = (
"You are an expert agricultural plant pathologist. Inspect the crop leaf image "
"and return a strict JSON object containing visual_evidence, lesion_boxes "
"[ymin, xmin, ymax, xmax] (0-1000 scale), species, diagnosis, and immediate_summary."
)
img = Image.open("sample_leaf.jpg").convert("RGB")
messages = [
{"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]},
{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "Diagnose this crop leaf and locate any pathological symptoms."}]},
]
input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(img, input_text, add_special_tokens=False, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=512, use_cache=True, do_sample=False)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip()
print(json.dumps(json.loads(response), indent=2))