Literary Sentiment Ordinal BERT — Medium

This model is BERT-medium (41.7M parameters) fine-tuned for sentence-level sentiment analysis of English literary text using ten ordered sentiment levels (1 = most negative, 10 = most positive). It is one of five student configurations trained on Gemma 4 31B-generated pseudo-labels for literary sentences.

The training objective is cumulative-logit ordinal regression, rather than conventional three-class classification. The scalar score is the expected ordinal rating.

• Original pretrained encoder: google/bert_uncased_L-8_H-512_A-8
• Trained checkpoint: BERT-medium; maximum training input length: 128 tokens
• Scores: 1–3 negative, 4–6 near-neutral, 7–10 positive (descriptive interpretation)
• Checkpoint selected by minimum validation MAE, with Hemingway annotations kept out of training and selection

On the independent human-annotated Hemingway evaluation (n=1,923), this model achieved MAE 0.752, Kendall τ 0.489, Spearman ρ 0.568 and accuracy within ±1 0.808. On 305 reference-defined non-neutral sentences, it achieved MAE 0.998 and accuracy within ±1 0.675.

Important: this is a custom ordinal head. The model produces nine cumulative logits, not ten class logits. Do not use AutoModelForSequenceClassification or a default pipeline("sentiment-analysis") to interpret the outputs. Use the AutoModel example below; its trust_remote_code=True loads modeling_ordinal_bert.py from this repository. Review that file before running repository code.

Quick start

Install pip install torch transformers and use:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "susurofu/literary-bert-en-medium"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()

sentences = ["A wonderful day filled with joy.", "Everything was lost."]
inputs = tokenizer(sentences, padding=True, truncation=True,
                   max_length=128, return_tensors="pt")
with torch.inference_mode():
    scores = 1 + model(**inputs).logits.sigmoid().sum(dim=-1)
labels = scores.round().clamp(1, 10).int()
print(scores.tolist())  # continuous expected score in [1, 10]
print(labels.tolist())  # nearest integer rating (1..10)

torch.round follows the notebook's numpy.rint nearest-integer convention (half-to-even); typically the scores are noninteger. For GPU inference, move both model and tokenized tensors to the same device. See example_inference.py for a standalone version. The nine output logits represent P(Y > k) after sigmoid for thresholds k=1,...,9 and must not be passed through softmax/argmax.

Training data and supervision

• 10,889 cleaned literary sentences from Project Gutenberg; an author/work sampling procedure and removal of short or structural passages were applied.
• Labels assigned using Gemma 4 31B with each sentence annotated independently on the 1–10 scale. The teacher returned a JSON field Sentiment_score; hard integer pseudo-labels were used for training.
• Stratified 80/10/10 train / validation / LLM-labeled test split: 8,711 / 1,089 / 1,089. These splits are distinct from the independent human-labeled test set.
• The independent human evaluation consists of 1,923 sentences from Hemingway's The Old Man and the Sea, with reference scores averaged across two annotators. Hemingway was not used for model fine-tuning or checkpoint selection.
• Teacher labels are imbalanced: 41.5% received score 5, and 57.9% received scores 4–6. The complete label frequencies (before splitting) are:

Architecture and training

The encoder produces a pooled representation, mapped to one latent sentiment score s(x). Nine trainable ordered thresholds θ₁ < ... < θ₉ are parameterized by a first threshold and cumulative softplus increments. The model returns:

logit_k = s(x) − θ_k, for k=1,...,9.

Training minimizes binary cross-entropy against nine cumulative binary targets 1[label > k]. At inference, E[Y|x] = 1 + Σ sigmoid(logit_k); the rounded rating is clipped to [1,10]. This is hard-label pseudo-labeling, not soft-logit distillation.

Evaluation: independent human-annotated Hemingway corpus

All sentences (n = 1,923). Each row reports metrics computed against the average of two human annotators. The table includes the teacher and external baseline results for context; those models are not included in this repository. The focus model is shown in bold. All numbers are from the supplied evaluation CSV (rounded to three decimals).

Full 3-class precision by predicted sentiment class (all sentences)

Non-neutral sentences (n = 305). This subset is selected using human reference scores below 4 or above 6. It measures performance on already-identified non-neutral sentences, not independent detection of such sentences.

Full 3-class precision by predicted sentiment class (non-neutral subset)

Metric definitions and interpretation

• MAE is computed from the original continuous model predictions and original averaged human scores (lower is better). Kendall's τ, Spearman's ρ, and Pearson's r use these unrounded scores.
• Accuracy (±1) is the proportion of samples with absolute error ≤ 1 in the original numerical scale.
• Precision (±1) is a custom legacy measure: precision is calculated separately at each observed distinct predicted numerical value, counting predictions within ±1 of the reference as correct, then macro-averaged over those distinct values. On continuous predictions it is not conventional classification precision and may be sensitive to floating-point uniqueness. Interpret cautiously.
• Three-class precision and accuracy are separate from ±1 metrics. Continuous predictions and averaged human scores are first rounded to integers using floor(score + 0.5) (half-up); scores 1–3 are negative, 4–6 neutral, and 7–10 positive. Macro precision averages the three per-class precision values, giving zero to classes without any predicted members.
• Caution: The human test corpus is very neutral-skewed: 84.1% of reference scores are in [4,6]. A constant-neutral predictor achieves overall MAE 0.597, better than all models; its correlations are undefined. Therefore, use multiple metrics and inspect the non-neutral subset rather than ranking models by overall MAE alone. The constant-neutral row predicts a constant score 5.
• Values across the two evaluation tables are not comparable without accounting for the different reference-selected subsets. RoBERTa results reflect their original numerical rescaling, rather than an ordinal model trained on 1–10 labels.

Performance on the held-out LLM-labeled test split (n = 1,089)

The following results measure agreement with the teacher-generated labels, not with Hemingway's human annotations.

Intended use

Research on literary sentence sentiment, relative valence patterns, and exploratory Digital Humanities analysis of narrative text. Intended for English. Sentiment scoring is inherently interpretation-dependent; examine outputs in textual context before drawing literary conclusions. This model is not a general-purpose classifier for clinical, legal, or other high-stakes decision-making.

Limitations

• Models were trained using automatically generated labels, which may inherit teacher biases (including topic-associated polarity).
• Single-sentence annotation and modeling omit wider narrative context, irony, and discourse-level sentiment shifts.
• The human-annotated evaluation covers one literary work; results do not guarantee similar performance across genres, dates, or authors.
• The training setup used one fixed seed. Reported small differences between model sizes have no uncertainty estimates.
• The 1–10 scale is ordinal: predicted values should not be treated as calibrated probabilities, and the expected score need not match an individual reader's interpretation.
• Faster inference than the teacher is plausible given model size, but has not been benchmarked in the corresponding study.

Related model variants

• BERT-large: susurofu/literary-bert-en-large (340M)
• BERT-base: susurofu/literary-bert-en-base (110M)
• BERT-medium: susurofu/literary-bert-en-medium (41.7M)
• BERT-small: susurofu/literary-bert-en-small (29.1M)
• BERT-mini: susurofu/literary-bert-en-mini (11.3M)

BERT-tiny was trained during development but is not released because it converged to constant neutral predictions. The constant-neutral baseline remains in the evaluation tables for context.

License and Attribution

This model is released under the Apache License 2.0. It is fine-tuned from Google's BERT architecture (Apache 2.0) using literary sentences from Project Gutenberg annotated by Gemma 4 31B (Apache 2.0). The model may be used, modified, and redistributed under the terms of the Apache License 2.0.

Citation

@misc{suslov2026distillingliterarysentimentllm,
      title={Distilling Literary Sentiment from LLM Annotations into an Ordinal BERT Model}, 
      author={Artem Suslov and Takehiro Hashimoto},
      year={2026},
      eprint={2611.02985},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2611.02985}, 
}