GPT-2 Reward Model — Anthropic HH-RLHF

A GPT-2-based reward model trained on the Anthropic HH-RLHF preference dataset using the Bradley–Terry preference-learning objective.

The model takes a response and produces a scalar reward. Given two responses to the same prompt, the reward model is trained to assign a higher reward to the human-preferred response.

Hugging Face repository: philomath-1209/gpt2-reward_model_hh-rlhf

---

Model Summary

This project implements a reward model from first principles rather than using a pre-built reward-model training framework.

The architecture consists of:

Input response
      ↓
GPT-2 tokenizer
      ↓
GPT-2 Transformer backbone
      ↓
Final non-padding token representation
      ↓
Linear reward head
      ↓
Scalar reward

For a pair of responses:

chosen response   ──→ reward_w
rejected response ──→ reward_l

                 reward_w - reward_l
                         ↓
                  Bradley–Terry
                     objective

The model does not apply a sigmoid to the final scalar reward. The Bradley–Terry objective operates directly on the reward difference.

---

Training Dataset

The model was trained using:

Dataset: Anthropic/hh-rlhf

The original training split was divided deterministically into training and validation subsets using a fixed seed.

The original test split was kept separate and was not used for model selection.

The test set was used only for final evaluation.

---

Preference Learning

Each training example contains two responses:

• chosen — the preferred response
• rejected — the non-preferred response

The reward model produces:

$$ r_w = r(x, y_w) $$

for the chosen response and:

$$ r_l = r(x, y_l) $$

for the rejected response.

The Bradley–Terry model defines the probability that the chosen response is preferred as:

$$ P(y_w \succ y_l)

\sigma(r_w-r_l) $$

where:

$$ \sigma(z)=\frac{1}{1+e^{-z}} $$

The training objective is:

$$ \mathcal{L}

-\log\sigma(r_w-r_l) $$

The implementation uses logsigmoid for numerical stability.

---

Model Architecture

GPT-2 Backbone

The model uses the pretrained GPT-2 Transformer architecture as the representation backbone.

The Transformer produces a contextual representation for every token:

$$ H = (h_1,h_2,\ldots,h_T) $$

where (T) is the sequence length.

Final Non-Padding Token

Because the model needs one representation for the entire response, the hidden state corresponding to the final non-padding token is selected.

For an attention mask (m):

$$ t_{\text{final}}

\sum_i m_i - 1 $$

and:

$$ h_{\text{final}}

H_{t_{\text{final}}} $$

Reward Head

The selected representation is passed through a linear layer:

$$ r = Wh_{\text{final}} + b $$

producing one scalar reward.

No sigmoid is applied to this scalar.

---

Training Configuration

The final full-dataset experiment used:

AdamW was used with separate parameter groups so that bias and normalization parameters were excluded from weight decay.

---

Checkpoint Selection

A checkpoint was saved after each completed epoch.

Two checkpoint types were maintained:

• latest_model.pt — most recent completed epoch
• best_model.pt — checkpoint with the best validation accuracy

The final model used for test evaluation and export was:

Epoch 4 — best validation checkpoint

Epoch 5 did not improve validation accuracy.

---

Results

Best Validation Checkpoint

Final Test Evaluation

The untouched test split contained 8,552 examples.

The validation and test accuracies differ by only 0.13 percentage points.

Training Metrics at Epoch 5

For reference, the final configured epoch produced:

The Epoch-4 checkpoint remained the best checkpoint.

---

Important Training Note

During the final experiment, an implementation issue caused the scheduler's configured number of training steps and the actual global step count to become inconsistent.

The scheduler was configured for:

4,523 batches × 5 epochs = 22,615 steps

while the recorded final global step reached 27,138.

Consequently, the learning rate reached zero during the final epoch.

This did not change the selected checkpoint: the best model was Epoch 4, before this issue affected the final epoch.

The scheduler/resume implementation has subsequently been identified for correction in the training repository so that future runs explicitly verify that expected and actual optimizer steps agree.

The published model itself is the validated Epoch-4 checkpoint.

---

Export Verification

The final checkpoint was exported to a Hugging Face Transformers-compatible model.

The exported model contains:

• config.json
• custom configuration_reward.py
• custom modeling_reward.py
• model.safetensors
• GPT-2 tokenizer files

The exported model was loaded independently and compared against the original PyTorch checkpoint.

Round-trip verification

Examples tested: 20
Maximum reward difference: 0.0000000000

PASS: Exported model matches the original checkpoint.

This verifies that the exported model reproduces the original checkpoint's reward outputs for the tested examples.

---

Loading the Model

The model can be loaded through the Hugging Face Transformers AutoModel interface.

from transformers import AutoTokenizer, AutoModel

model_id = "philomath-1209/gpt2-reward_model_hh-rlhf"

tokenizer = AutoTokenizer.from_pretrained(
    model_id
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
)

Because this repository contains a custom Transformers model implementation, trust_remote_code=True is required when loading the custom architecture.

Only enable trust_remote_code=True for repositories whose custom code you trust and have inspected.

---

Scoring a Response

import torch

text = "Your response goes here."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

with torch.no_grad():
    reward = model(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
    )

print("Reward:", reward.item())

The returned scalar is the model's learned reward for the response.

---

Comparing Two Responses

import torch

chosen = "Preferred response."
rejected = "Less preferred response."

chosen_inputs = tokenizer(
    chosen,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

rejected_inputs = tokenizer(
    rejected,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

with torch.no_grad():

    chosen_reward = model(
        input_ids=chosen_inputs["input_ids"],
        attention_mask=chosen_inputs["attention_mask"],
    )

    rejected_reward = model(
        input_ids=rejected_inputs["input_ids"],
        attention_mask=rejected_inputs["attention_mask"],
    )

print("Chosen reward:", chosen_reward.item())
print("Rejected reward:", rejected_reward.item())

A higher reward indicates that the model assigns greater preference to that response.

---

Interpreting the Reward

The absolute magnitude of the reward should not be interpreted as a universal quality score.

The model is trained primarily through relative comparisons:

$$ r(y_w) > r(y_l) $$

The difference between two rewards is therefore more meaningful than the absolute value of either reward.

For example:

Response A → 2.3
Response B → 1.1

indicates that the model prefers A over B according to its learned reward function.

It does not mean that A has some universally defined "quality score" of 2.3.

---

Limitations

This model should be interpreted as a learned preference model for the training distribution and task on which it was trained.

The reported:

62.61%

is pairwise preference accuracy on the held-out HH-RLHF test set.

It should not be interpreted as:

• universal human-preference accuracy
• general-purpose response quality
• factual correctness
• safety correctness
• alignment across arbitrary domains
• a calibrated absolute quality score

The model can prefer an individual rejected response over its corresponding chosen response. Pairwise accuracy below 100% is expected for a learned reward model.

The model was trained on a specific preference dataset, and its learned reward function may inherit the biases, limitations, and distribution characteristics of that dataset.

---

Intended Use

This model is primarily intended for:

• studying reward modeling
• preference-learning experiments
• understanding Bradley–Terry reward objectives
• experimenting with RLHF pipelines
• ranking candidate responses
• educational and research purposes

It is not intended to be treated as a universal evaluator of language-model outputs.

---

Training Repository

The complete implementation, including:

• dataset processing
• reward-model architecture
• Bradley–Terry loss
• training loop
• evaluation
• checkpointing
• resume logic
• export utilities
• verification scripts
• configuration

is available in the corresponding GitHub repository.

---

Reproducibility

The project uses a fixed random seed and explicit configuration files.

The full workflow is:

Dataset
   ↓
Train / Validation split
   ↓
Reward model
   ↓
Bradley–Terry training
   ↓
Checkpoint selection
   ↓
Test evaluation
   ↓
Hugging Face export
   ↓
Export verification

The training repository contains the exact configuration and commands used during development and final evaluation.

---

License and Dataset Attribution

This model is built using the Anthropic/hh-rlhf dataset and the GPT-2 architecture.

Users should review the original dataset and GPT-2 licensing and usage terms before redistributing or deploying derivative systems.

---

Citation

If you use this implementation or model in your work, please refer to the associated GitHub repository and Hugging Face model repository.

Hugging Face: philomath-1209/gpt2-reward_model_hh-rlhf