Social Group Stance Classification (NLI)

A DeBERTa-v3-base NLI model fine-tuned to classify the author's stance toward a social group in a political text as positive, negative, or neutral.

This model is part of the group-appeal-detector package, which also provides group mention detection and mention clustering.

Model Details

Base model: MoritzLaurer/deberta-v3-base-zeroshot-v2.0
Task: Stance Classification (NLI-based)
Labels: positive, negative, neutral
Training data: 5,000 manually annotated sentences from UK House of Commons parliamentary debates (2010–2019), with the negative class oversampled by adding synthetic paraphrases for half of all negative social group appeals

How It Works

For each detected group mention, three hypotheses are formulated — one per stance class. The model chooses the class with the largest entailment probability as the predicted stance.

Performance

Cross-validated performance (95% confidence intervals based on the estimated standard error across folds in brackets):

Metric	Negative	Neutral	Positive	Macro-Avg.
F1	0.76 [0.72, 0.80]	0.80 [0.78, 0.81]	0.89 [0.89, 0.89]	0.81 [0.80, 0.83]
Precision	0.85 [0.77, 0.94]	0.81 [0.79, 0.84]	0.87 [0.86, 0.88]	0.85 [0.82, 0.87]
Recall	0.70 [0.62, 0.77]	0.78 [0.76, 0.80]	0.91 [0.89, 0.92]	0.79 [0.77, 0.82]

Usage

Via `group-appeal-detector` package (recommended)

pip install group-appeal-detector

from group_appeal_detector import GroupAppealDetector

detector = GroupAppealDetector(device="cpu")

sentence = "We must protect the rights of farmers."
target_group = "farmers"
result = detector.classify_stance(sentence, target_group)
print(result["predicted_stance"], result["stance_probs"])

For batch processing:

pairs = [
    ("We must protect the rights of farmers.", "farmers"),
    ("We do a lot for elderly people.", "elderly people"),
]
results_df = detector.classify_stance_batch(pairs, batch_size=8, as_df=True)

Direct usage with Transformers

from transformers import pipeline

pipe = pipeline(
    "zero-shot-classification",
    model="maxwlnd/socialgroup_stance_classification_nli",
)

sentence = "We must protect the rights of farmers."
target_group = "farmers"

hypotheses = [
    f"The text is positive towards {target_group}.",
    f"The text is negative towards {target_group}.",
    f"The text is neutral, or contains no stance, towards {target_group}.",
]

result = pipe(sentence, candidate_labels=hypotheses)
print(result["labels"][0], result["scores"][0])

Related Models

This model is one of three models in the group appeal detection pipeline:

Model	Task
`maxwlnd/roberta_group_mention_detector`	Detect social group mentions
`maxwlnd/socialgroup_stance_classification_nli`	Classify stance toward a group as positive, negative, or neutral (this model)
`maxwlnd/cl_mention_embedding`	Embed mentions for clustering into qualitative categories

Conceptual Background

The definition of a social group appeal used for annotating the training data is inspired by Lena Maria Huber and Alona O. Dolinsky and Will Horne, Alona O. Dolinsky and Lena Maria Huber.

A group appeal is an intentional act by a political actor that associates them with a social group in a supportive or critical manner. This model classifies whether the author's expressed stance toward the mentioned group is positive, negative, or neutral.

License

MIT

maxwlnd/socialgroup_stance_classification_nli