evalexplorer-classify-gemma-4-e4b-sft
Classifies an international development evaluation report from its first pages into a JSON object with evaluation_approach, evaluation_type, temporality, themes and countries. One of the runs documented in baobabtech/evalexplorer-classify-experiments; code in code/.
| Base model | unsloth/gemma-4-E4B-it |
| Method | SFT |
| Data | baobabtech/evalexplorer-data, config classify_codes, 1148 training documents |
| Hardware | gpu |
| Training time | 45.3 min |
| Run report | gemma-4-e4b-sft--test |
Training
LoRA SFT: r=16, alpha=16, all linear language layers, bf16, learning rate 0.0002, 2 epochs, batch 2 × 8 accumulation, loss on the answer only.
Scores
| Metric | Test (134 documents) |
|---|---|
| JSON valid | 1.000 |
| Exact match (all five fields) | 0.261 |
| Mean field score | 0.830 |
| evaluation_approach accuracy | 0.799 |
| evaluation_type accuracy | 0.813 |
| temporality accuracy | 0.806 |
| themes micro F1 | 0.840 |
| countries micro F1 | 0.805 |
Greedy decoding, thinking off. mean_field_score is the per-document mean of the five field scores (1/0 for the single-code fields, F1 for the lists). The model was trained on, and is scored here against, the EvalExplorer ingestion pipeline's LLM labels, which no person has reviewed, so these numbers measure agreement with that pipeline, not correctness. The run report also scores the same answers against an independent GLM-5.3-Flash relabelling the model never saw. This is an exploratory model; the intended next version is trained on the GLM labels.
Usage
from unsloth import FastLanguageModel, FastModel
loader = FastLanguageModel if "lfm" in "unsloth/gemma-4-E4B-it".lower() else FastModel
model, processor = loader.from_pretrained("unsloth/gemma-4-E4B-it", max_seq_length=8192, load_in_4bit=False)
from peft import PeftModel
model = PeftModel.from_pretrained(model, "baobabtech/evalexplorer-classify-gemma-4-e4b-sft")
loader.for_inference(model)
text = processor.apply_chat_template(prompt_messages, tokenize=False, add_generation_prompt=True,
enable_thinking=False) # prompt_messages from the dataset's `prompt` column