DistilBERT-SarcOji-KD-v2
distilbert student fine-tuned on SarcOji with sarcasm-aware emoji embeddings injected
directly into the token-embedding matrix, distilled from Vandita/Bert-finetuned-Sarc.
Results
| split | accuracy | precision | recall | F1 | MCC | ROC-AUC |
|---|---|---|---|---|---|---|
| validation | 0.8832 | 0.8550 | 0.8430 | 0.8490 | 0.7538 | 0.9531 |
| SarcOjiTest1 | 0.6606 | 0.7545 | 0.5257 | 0.6196 | 0.3482 | 0.7489 |
| SarcOjiTest2 | 0.7275 | 0.4455 | 0.3721 | 0.4055 | 0.2321 | 0.6637 |
SarcOjiTest2 is ~75/25 negative-skewed, so its accuracy is not comparable to Test1's — MCC and ROC-AUC are the meaningful columns there.
Distillation
Hinton et al. (2015): L = α·T²·KL(teacher‖student) + (1−α)·CE, with T = 3,
α = 0.5. The T² factor is applied, so soft-target gradients do not shrink as T
rises.
Emoji handling
1444 emoji tokens added to the tokenizer (vocab
30522 → 31959). Each new row of the
768-d embedding matrix was initialised from a 300-d sarcasm-aware
emoji vector using concat-mean. Remaining dimensions hold the mean of the
original vocabulary embeddings rather than zeros, so new tokens are not anchored to the
origin.
Because the emoji are part of the vocabulary, this is a standard HuggingFace model — no custom architecture, no custom loader.
Usage
from transformers import AutoModelForSequenceClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Vandita/DistilBERT-SarcOji-KD-v2")
model = AutoModelForSequenceClassification.from_pretrained("Vandita/DistilBERT-SarcOji-KD-v2")
Data
SarcOji, 80:20 stratified split. Duplicates retained; identical texts were kept on one side of the split so no text appears in both train and validation. Evaluated on SarcOjiTest1 and SarcOjiTest2 in full.