A LoRA adapter that teaches Mistral-7B-Instruct-v0.3 context-grounded commonsense reasoning in Farsi
This adapter was fine-tuned on the training split of FarSense, a benchmark for commonsense reasoning in Farsi (Persian), for the paper FarSense: A Comprehensive Commonsense Benchmark and Evaluation Framework for the Farsi Language (IJCNLP-AACL 2025). It is one of six adapters released with the paper; all of them, the dataset and the code are listed in the FarSense collection and on GitHub.
Each FarSense item gives the model an everyday scenario in Farsi and asks for one JSON answer: True/False judgements, multiple choice, cloze completion, cause→effect, counterfactual and "why" questions.
Fine-tuning raises the overall score of Mistral-7B-Instruct-v0.3 by +0.153 (0.496 → 0.649).
| Metric | Base (zero-shot) | FarSense LoRA | Δ |
|---|---|---|---|
| T/F-True accuracy | 0.743 | 0.919 | +0.176 |
| T/F-False accuracy | 0.910 | 0.957 | +0.048 |
| Multiple-choice accuracy | 0.638 | 0.962 | +0.324 |
| Cloze accuracy | 0.171 | 0.429 | +0.257 |
| Cause→Effect (text score) | 0.365 | 0.444 | +0.079 |
| Counterfactual (text score) | 0.372 | 0.444 | +0.072 |
| Explanation (text score) | 0.319 | 0.421 | +0.101 |
| Cloze (text score) | 0.451 | 0.619 | +0.168 |
| Classification | 0.615 | 0.817 | +0.201 |
| Free-form | 0.377 | 0.482 | +0.105 |
| Overall | 0.496 | 0.649 | +0.153 |
Accuracies are exact match. Text scores average BLEU-4, ROUGE-L F1, BERTScore F1 and embedding cosine similarity. Classification averages the four accuracies, Free-form the four text scores, and Overall the two (Section 4.2 of the paper). Scores are on the 1,470 test items (210 scenarios) with the sampling settings below.
import torch
from datasets import load_dataset
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "mistralai/Mistral-7B-Instruct-v0.3"
adapter_id = "Kamyar-zeinalipour/FarSense-Mistral-7B-Instruct-v0.3-LoRA"
tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id).merge_and_unload()
item = load_dataset("Kamyar-zeinalipour/CS_Farsi", split="test")[0]
chat = tokenizer.apply_chat_template(
[{"role": "user", "content": item["prompt"]}], tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(chat, return_tensors="pt").to(model.device)
output = model.generate(
**inputs, max_new_tokens=256, do_sample=True,
temperature=0.8, top_p=0.95, top_k=50, repetition_penalty=1.1,
)
print("model:", tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
print("gold: ", item["assistant"])
The base model is gated: accept its terms on its model page and
log in with huggingface-cli login first. To score a full run, use the evaluation scripts in the
GitHub repository.
The adapter is 1.2 GB because LoRA was also applied to embed_tokens and lm_head, and PEFT
stores those layers with the adapter.
| Base model | mistralai/Mistral-7B-Instruct-v0.3 (7B) |
| Data | FarSense / CS_Farsi train split: 4,000 scenarios, 28,000 items |
| Method | LoRA, r = 128, α = 256, dropout 0.1, on q/k/v/o, gate/up/down, embed_tokens, lm_head |
| Schedule | 3 epochs, AdamW, learning rate 1e-4, cosine decay, no warm-up, weight decay 1e-4 |
| Batch | 4 sequences per GPU, 2 gradient-accumulation steps, packed to 2,048 tokens |
| Precision | bf16, DeepSpeed ZeRO-3, Flash Attention 2, gradient clipping 1.0, seed 100 |
| Hardware | 3 × NVIDIA RTX A6000 (48 GB), 9 h 37 min |
| Decoding (evaluation) | temperature 0.8, top-p 0.95, top-k 50, repetition penalty 1.1, ≤ 256 new tokens |
The training script and configuration are in the GitHub repository.
| Base model | Params | Adapter | Overall (base → fine-tuned) |
|---|---|---|---|
| Llama-3.2-1B-Instruct | 1B | FarSense-Llama-3.2-1B-Instruct-LoRA | 0.227 → 0.450 |
| Llama-3.2-3B-Instruct | 3B | FarSense-Llama-3.2-3B-Instruct-LoRA | 0.449 → 0.529 |
| Llama-3.1-8B-Instruct | 8B | FarSense-Llama-3.1-8B-Instruct-LoRA | 0.598 → 0.642 |
| Mistral-7B-Instruct-v0.3 | 7B | FarSense-Mistral-7B-Instruct-v0.3-LoRA (this model) | 0.496 → 0.649 |
| Mistral-Nemo-Instruct-2407 | 12B | FarSense-Mistral-Nemo-Instruct-2407-LoRA | 0.604 → 0.654 |
| Mistral-Small-24B-Instruct-2501 | 24B | FarSense-Mistral-Small-24B-Instruct-2501-LoRA | 0.627 → 0.672 |
@inproceedings{zeinalipour-etal-2025-farsense,
title = "{F}ar{S}ense: A Comprehensive Commonsense Benchmark and Evaluation Framework for the {F}arsi Language",
author = "Zeinalipour, Kamyar and
Jamshidi, Neda and
Hejazi, Seyedehbahareh and
Maggini, Marco and
Bianchini, Monica and
Paoletti, Simone and
Gori, Marco",
booktitle = "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the {A}sia-Pacific Chapter of the {A}ssociation for {C}omputational {L}inguistics",
month = dec,
year = "2025",
address = "Mumbai, India",
publisher = "The Asian Federation of Natural Language Processing and The Association for Computational Linguistics",
url = "https://aclanthology.org/2025.ijcnlp-long.187/",
doi = "10.18653/v1/2025.ijcnlp-long.187",
pages = "3529--3599",
}
This adapter is a derivative of Mistral-7B-Instruct-v0.3 and is released under the
base model's license (apache-2.0). The FarSense data is CC BY-SA 4.0; the code is Apache-2.0.
mistralai/Mistral-7B-Instruct-v0.3