Hugging Face | GitHub | LinkedIn | WhatsApp | Portfolio | Medium Blog

Author: Md. Emon Hasan

liar-roberta

A binary classifier that labels a short political statement as REAL or FAKE. Full fine-tune of roberta-base on LIAR, the PolitiFact fact-check corpus.

On the LIAR test split it reaches 64.5% accuracy and 0.6351 macro-F1, clearly ahead of a TF-IDF + logistic regression baseline trained on the same data (0.6025 macro-F1). Every number on this page comes from the single run that produced these weights; the raw values are in results.json.

from transformers import pipeline
clf = pipeline("text-classification", model="emonHasan01/liar-roberta")
clf("The moon landing was filmed in a Hollywood studio.")

Label scheme

LIAR ships six truthfulness classes, collapsed here into two:

Worth flagging: LIAR's integer label order is not sorted by truthfulness (4 = barely-true and 5 = pants-fire are the two least truthful, not the most). A fair amount of published code treats them as the truthful end and inverts the whole task. This model resolves the mapping from the dataset's label names, never indices.

The collapse leaves the data imbalanced — 460 REAL against 823 FAKE in the test split — so training used class-weighted cross-entropy and the checkpoint was chosen on macro-F1 rather than accuracy.

Results

The majority-class row is there on purpose. Accuracy alone on this dataset says very little, since always answering FAKE already scores 0.6415.

Where it succeeds and fails

Accuracy across the six original LIAR classes, before the binary collapse:

The ends of the scale are separable; the middle is not. half-true and mostly-true differ by an editorial judgement that is largely unrecoverable from 18 words of text, and the model's scores reflect that.

Most confident mistakes

Of 456 errors, 150 are real statements flagged as fake and 306 are fake statements let through.

Choosing a threshold

P(FAKE) is exposed, so the 0.5 cutoff is a starting point rather than a verdict:

At threshold 0.65 precision reaches 0.811, with recall falling to 0.385.

Is the confidence meaningful?

Expected calibration error is 0.042 over 10 bins. Points below the diagonal mean overconfidence: when the model says 0.9 it is right less often than 90% of the time. Treat the score as a ranking signal, not as a probability of being lied to.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("emonHasan01/liar-roberta")
model = AutoModelForSequenceClassification.from_pretrained("emonHasan01/liar-roberta")

inputs = tok("Your statement here", return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]
print("P(REAL)", probs[0].item(), "P(FAKE)", probs[1].item())

Longer documents

The model saw statements of about 22 tokens and has a 128-token window. Feeding a whole article truncates it. Chunk and average instead:

def classify_long(text, tok, model, chunk_words=60):
    w = text.split()
    chunks = [" ".join(w[i:i+chunk_words]) for i in range(0, len(w), chunk_words)] or [text]
    enc = tok(chunks[:32], return_tensors="pt", truncation=True, padding=True, max_length=128)
    with torch.no_grad():
        p = model(**enc).logits.softmax(-1).mean(0)
    return {"REAL": p[0].item(), "FAKE": p[1].item()}

Results on full articles are out of distribution either way — see Limitations.

Training

Statement length in the training set: median 22 tokens, 95th percentile 40, so a 128-token window truncates almost nothing.

Limitations

Please read this section before using the model for anything that matters.

• It does not check facts. There is no retrieval and no knowledge base. The model learns lexical and stylistic patterns that correlate with a PolitiFact verdict. A false claim written in the register of a true one gets through, which the confident-mistakes table above shows directly.
• The domain is narrow. Short US political statements, roughly 2007–2016, selected for fact-checking by PolitiFact. It has not learned to judge full news articles, scientific claims, health advice, or non-US or non-political content, and accuracy there is unmeasured and probably much worse.
• The binary collapse discards information. Grouping half-true with pants-fire throws away most of the distinction the annotators recorded.
• It inherits PolitiFact's selection. Which speakers and claims get fact-checked at all is an editorial decision, absorbed here along with the labels.
• It is overconfident (ECE 0.042). Do not read the softmax score as a probability.
• Temporal drift. Names, events, and political vocabulary have moved on since the corpus was collected.
• The ceiling is low, and that is expected. LIAR is a hard benchmark and published binary results sit broadly in this range. A very high accuracy on LIAR usually means a label-mapping bug or a leak rather than a better model.

Do not use this as an automated arbiter of truth, for moderation at scale, or in any decision affecting a person without a human reading the output.

Citation

@inproceedings{wang2017liar,
  title     = {"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection},
  author    = {Wang, William Yang},
  booktitle = {Proceedings of ACL},
  year      = {2017}
}

Reproducing

Everything on this page — training, both baselines, all five charts and this card — comes from one Colab notebook run on a Tesla T4. No metric was typed in by hand.