Qwen2.5-0.5B grounded reward model

A sequence-classification reward model for the grounded QA preference dataset. It is initialized from the SFT policy, with a new scalar score head, and trained with TRL RewardTrainer on the Bradley-Terry loss: the score of the chosen reply should exceed the score of the rejected reply.

The head is merged and saved with the backbone, so this repository loads with AutoModelForSequenceClassification. The score is a single logit. Higher means the reply is closer to the preference rules (right span when the passage has the answer, abstention when it does not).

The training data is derived from SQuAD 2.0 and is licensed CC BY-SA 4.0. These weights stay under the base model's Apache 2.0 license.

How to score a reply

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_id = "saitejaalasyam/qwen2.5-0.5b-grounded-reward"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
messages = [
    {"role": "system", "content": "You answer a question using only the passage in the user message. Reply with a short answer taken from the passage. If the passage does not contain the answer, reply with exactly: The passage does not say."},
    {"role": "user", "content": "Question: When was Ada Lovelace born?\n\nReply with only the short answer. If the passage does not contain it, reply with exactly: The passage does not say.\n\nPassage:\nAda Lovelace was born in 1815."},
    {"role": "assistant", "content": "1815"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False)
batch = tokenizer(text, return_tensors="pt").to(model.device)
print(model(**batch).logits.item())

Evaluation

eval_accuracy during training is the fraction of development pairs where the chosen reply scores higher than the rejected one. That number is not a checkpoint-selection metric (save_strategy: no), and it is not the final test score. shortcut.json in the dataset repository compares this model with a length-only rule and an abstention-only rule on each pair type, including length-matched pairs, length-equalized pairs, abstention paraphrases, and clipped plausible wrong answers. The policy comparison below uses this model as the judge.

Config: protocol.yaml.

Limitations

The reward model can prefer surface cues in this dataset, including the abstention sentence itself. It is a training signal for this task, not a general preference model.