Fakeddit Fake-News Classifier (distilbert-base-uncased)

Fine-tuned on a balanced subsample of Fakeddit for binary (real/fake) headline classification, with validation-based checkpoint selection and post-hoc temperature-scaled calibration.

This model classifies headline style, not claim veracity. It is not a general-purpose fact-checker. See Limitations below.

Training data

• Source: Fakeddit official train/validate/test TSVs, pooled and re-split (not directly comparable to papers using Fakeddit's own official test split).
• Balanced to 2500 examples per class.
• Split: 3500 train / 500 validation / 1000 test, stratified.

Methodology

• Candidates fine-tuned across 3 seeds each: bert-base-uncased, distilbert-base-uncased.
• Checkpoint selected by validation F1 (best epoch: 4), test set evaluated exactly once.
• Post-hoc temperature scaling (Guo et al., 2017) fit on validation logits. Temperature T = 1.1099.
• Test ECE: 0.0813 (raw) -> 0.0656 (calibrated).

Evaluation results (held-out test set, evaluated once)

Multi-seed comparison (mean +/- std across 3 seeds):

Limitations

• Not a general-purpose fact-checker: labels come from Fakeddit's distant-supervision scheme (subreddit of origin), not per-claim human fact-checking.
• Trained only on short English Reddit post titles; untested on articles, other languages, or claims postdating training data collection.
• 5000 examples is a small, rebalanced subsample of Fakeddit's full corpus.
• Calibration reduces average overconfidence, not per-example correctness.

How to use

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch, torch.nn.functional as F

model = AutoModelForSequenceClassification.from_pretrained("abhinav-29/fakeddit-bert-fake-news")
tokenizer = AutoTokenizer.from_pretrained("abhinav-29/fakeddit-bert-fake-news")
text = "Local city council approves new budget for public libraries"
enc = tokenizer(text, max_length=64, padding="max_length", truncation=True, return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits
probs = F.softmax(logits / 1.1099, dim=-1)  # calibrated confidence
print(probs)

Citation

Nakamura, K., Levy, S., & Wang, W. Y. (2020). r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection. Proceedings of LREC 2020.