Indic Climate Stance (Hindi / Bengali)

Stance of a Hindi or Bengali social-media comment (native script, Romanized or code-mixed) toward the claim "Climate change and global warming is a serious concern." -> Favour / Against / None.

Final configuration: avg_all (ensemble), calibration: null.

Intended use

Research and educational analysis of public climate discourse in Hindi and Bengali (comment sections, surveys). Not intended for moderation decisions about individuals, for inferring personal attributes, or for languages other than Hindi/Bengali/English mixes.

Training data

No official data existed; the training set was built from public English stance datasets (SemEval-2016 Task 6 climate target, GWSD, ClimateStance) translated to Hindi and Bengali with NLLB-200 (back-translation filtered by LaBSE cosine >= 0.75), Bengali headlines from Dhoroni, Romanized and code-mixed copies, and template-synthetic comments capped at 40 %. Train rows: 19278 (synthetic share 0.09); dev rows: 1402. See docs/datasheet.md in the repository for sources, licences and known biases.

Evaluation (hand-written gold set, test-only, 306 comments, 14 categories)

metric	value
Macro F1	0.7662
Accuracy	0.7712
Favour P / R / F1	0.840 / 0.573 / 0.681 (n=110)
Against P / R / F1	0.797 / 0.904 / 0.847 (n=104)
None P / R / F1	0.699 / 0.859 / 0.771 (n=92)
Macro F1 (bn)	0.7320 (n=153)
Macro F1 (hi)	0.8005 (n=153)

Challenge-suite failure rate (CheckList-style, 1084 tests): 0.209

category	fail rate	n
implicit	0.57	60
sarcasm	0.40	60
conspiracy	0.33	60
different_target	0.32	60
dir_negation	0.31	64
romanized	0.30	60
short	0.28	60
emoji	0.27	60
codemixed	0.20	60
negation	0.20	60
inv_roman_span	0.17	30
inv_typo	0.17	30
typos	0.17	60
neutral_topic	0.13	60
inv_url_append	0.10	30
inv_elongation	0.07	30
inv_emoji_append	0.03	30
inv_punct_strip	0.03	30
inv_mention_prepend	0.00	30
inv_whitespace	0.00	30
long	0.00	60
offtopic	0.00	60

Candidates compared (final selection)

candidate	dev_macro_f1	gold_macro_f1	challenge_fail_rate	score
avg_all	0.6856	0.7662	0.2094	0.6615
muril_pair_claim-en_weighted_ce	0.6830	0.7367	0.2196	0.6269
muril_pair_claim-en_weighted_ce+cal	0.6854	0.7304	0.2177	0.6215
weighted_dev+cal	0.6993	0.7313	0.2205	0.6211
avg_all+cal	0.7003	0.7398	0.2408	0.6194
weighted_dev	0.6991	0.7246	0.2113	0.6190
muril_pair_claim-auto_focal	0.6718	0.7324	0.2712	0.5968
tfidf_cnb	0.5781	0.7318	0.2869	0.5883
emb_bgem3_lr	0.6342	0.7193	0.2657	0.5864
stack_dev	0.6722	0.6836	0.2325	0.5674
hybrid_baseline	0.6376	0.6704	0.2998	0.5205

Accuracy vs cost (CPU)

model	gold_macro_f1	challenge_fail_rate	size_mb	cpu_ms_per_example
tfidf_cnb	0.732	0.287	19.142	0.171
hybrid_baseline (j)	0.670	0.300	23.130	118.207
muril_pair_claim-en_weighted_ce (fp32 cpu)	0.737	0.220	959.833	17.028
muril_pair_claim-en_weighted_ce (int8 dynamic, attention layers)	0.715	0.223	nan	14.319
ensemble (avg_all)	0.766	0.209	1938.823	196.774

Limitations and biases

Training text is mostly machine-translated from English tweets/news; style differs from real Hindi/Bengali comments (see the synthetic-to-handcrafted gap analysis in the report).
The gold set and lexicon were written by the project author; scores on it are indicative, not a substitute for real annotated data.
Sarcasm, implicit stance and stance toward a related-but-different target remain the hardest categories (see the robustness table).
Most configurations have one seed (4 GB GPU); for the three with three seeds, gold macro F1 varied by up to 0.18 between seeds, so small differences between models are not meaningful.
The model may reflect political framings present in the English source data (US/UK climate debate).

How to use

The model code lives in the GitHub repository; install it first:

git clone https://github.com/Aakash25-11-04/Climate_Stance_Indic.git
cd Climate_Stance_Indic
pip install -r requirements.txt && pip install -e .

Full final model (average of 4 members: two MuRIL models, TF-IDF + Complement NB, BGE-M3 + LR):

from huggingface_hub import snapshot_download
from stance.models.ensemble import EnsembleModel
root = snapshot_download('Aakash25-11-04/Climate_Stance_Indic')
model = EnsembleModel.from_dir(root)
texts, langs = ['जलवायु परिवर्तन एक गंभीर समस्या है', 'global warming sab jhooth hai'], ['hi', 'hi']
probs = model.combine(model.member_probas(texts, langs))   # columns: Favour, Against, None

Single transformer only (MuRIL, pair input; faster, gold macro F1 0.737):

from stance.models.transformer import TransformerStanceModel
m = TransformerStanceModel.load(root)
m.predict(['জলবায়ু পরিবর্তন একটি গুরুতর সমস্যা'], ['bn'])

From a clone of the repository: python scripts/predict.py --input comments.csv --output preds.csv --model final.

Citation

If you use this model or dataset pipeline, cite the repository and the source datasets listed in the datasheet.