Persian Paraphrase Identification with ParsBERT

A Persian Natural Language Processing model for identifying whether two Persian sentences have the same meaning (paraphrase) or not.

Task

This model performs binary sequence classification on pairs of Persian texts.

Class 0: Non-Paraphrase
Class 1: Paraphrase

The model predicts whether the two sentences convey the same meaning.

Base Model

The base pretrained model used in this project is:

HooshvareLab/bert-base-parsbert-uncased

The model was fine-tuned for Persian paraphrase identification using a binary classification head.

Dataset

The model was trained and evaluated on the PARSINLU QQP dataset.

Dataset source:

https://github.com/persiannlp/parsinlu

The dataset contains pairs of Persian questions/sentences with paraphrase labels.

Model Architecture

Base architecture: BERT / ParsBERT

Task: Sequence Classification

Number of classes: 2

Labels

ID	Label
0	Non-Paraphrase
1	Paraphrase

Fine-Tuning Strategy

Part of the pretrained BERT model was frozen during fine-tuning.

Frozen

Embedding layer
Encoder layers 0–7

Trainable

Encoder layers 8–11
Classification head

This strategy allows the upper transformer layers and classification head to adapt to the paraphrase identification task while keeping the lower-level pretrained representations fixed.

Training Configuration

Parameter	Value
Epochs	5
Learning Rate	2e-5
Train Batch Size	16
Evaluation Batch Size	32
Weight Decay	0.01
Maximum Sequence Length	128
Random Seed	42

Evaluation Metrics

The following metrics were used:

Accuracy
Precision
Recall
F1 Score

Test Results

Metric	Score
Accuracy	0.7636
Precision	0.6934
Recall	0.8189
F1	0.7510

Parameter Statistics

Parameter Type	Number
Total Parameters	162,842,882
Trainable Parameters	28,943,618
Frozen Parameters	133,899,264

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "amir1381sa/persian-paraphrase-bert"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text1 = "امروز هوا خیلی خوب است."
text2 = "امروز وضعیت آب و هوا بسیار مناسب است."

inputs = tokenizer(
    text1,
    text2,
    return_tensors="pt",
    truncation=True,
    max_length=128
)

with torch.no_grad():
    outputs = model(**inputs)

prediction = torch.argmax(outputs.logits, dim=-1).item()

print(model.config.id2label[prediction])

Output

The model returns one of the following labels:

Non-Paraphrase
Paraphrase

Repository Contents

This repository contains:

Fine-tuned ParsBERT model
Tokenizer
Model configuration
Test evaluation results
Training logs
Experiment configuration

Project Summary

This project demonstrates fine-tuning of a pretrained Persian BERT model for the task of Persian Text Paraphrase Identification.

The pretrained model was adapted to a binary classification task using the PARSINLU QQP dataset.

Reproducibility

Experiments were performed using:

Python
PyTorch
Hugging Face Transformers
Hugging Face Datasets
Scikit-learn

Random seed:

42