roberta-base-suicide-prediction-phr

This model is a fine-tuned version of roberta-base on this dataset sourced from Reddit. It achieves the following results on the evaluation/validation set:

• Loss: 0.1543
• Accuracy: 0.9652972367116438
• Recall: 0.966571403827834
• Precision: 0.9638169257340242
• F1: 0.9651921995935487

It achieves the following result on validation partition of this updated dataset

• Loss: 0.08761
• Accuracy: 0.97065
• Recall: 0.96652
• Precision: 0.97732
• F1: 0.97189

Model description

This model is a finetune of roberta-base to detect suicidal tendencies in a given text.

Training and evaluation data

• The dataset is sourced from Reddit.
• The dataset contains text with binary labels for suicide or non-suicide.
• The dataset was cleaned, and following steps were applied
  • Converted to lowercase
  • Removed numbers and special characters.
  • Removed URLs, Emojis and accented characters.
  • Removed any word contractions.
  • Remove any extra white spaces and any extra spaces after a single space.
  • Removed any consecutive characters repeated more than 3 times.
  • Tokenised the text, then lemmatized it and then removed the stopwords (excluding not).
• The cleaned dataset can be found here
• The evaluation set had ~23000 samples, while the training set had ~186k samples, i.e. a 80:10:10 (train:test:val) split.

Training procedure

• The model was trained on an RTXA5000 GPU.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Framework versions

• Transformers 4.31.0
• Pytorch 2.1.0+cu121
• Datasets 2.14.5
• Tokenizers 0.13.3