english_with_tiesA pairwise reward model for jokes: given a news item and two candidate jokes, it scores each so that the one humans preferred gets the higher reward. Trained for a thesis on humour generation with reinforcement learning; code and experiment logs live in the thesis repository under reward-model/.
english_with_tiesRewardModelWithTies (Bradley-Terry with a learned tie threshold; equally funny pairs count as ties)roberta-baseenglish| Folder | Contents |
|---|---|
best_trial/ |
The single-seed model of the best Optuna trial, chosen by its mean score over the cross-validation folds of the train split. The weights are those of its last fold model, trained on the other folds. Replaced every time a trial beats the previous best, so earlier bests are in the commit history. |
ensemble/ |
The final model: the winning configuration retrained on the whole train split once per seed, with rewards averaged across members. Use this one. |
best_trial/)| Hyperparameter | Value |
|---|---|
batch_size |
128 |
dropout |
0.1108 |
encoder_learning_rate |
1.559e-05 |
epochs |
4 |
freeze_strategy |
last_6 |
head_learning_rate |
0.0004479 |
layer_lr_decay |
0.9045 |
max_length |
256 |
micro_batch_size |
None |
model_name |
roberta-base |
scheduler_type |
cosine |
warmup_ratio |
0.05086 |
weight_decay |
0 |
ensemble/)| model | non_tied_preference_accuracy |
three_way_accuracy |
tie_recall |
tie_precision |
eval_loss |
mean_margin |
|---|---|---|---|---|---|---|
RewardModelWithTies_seed_42 |
0.5518 | 0.4482 | 0.002299 | 1 | 1.043 | 0.03508 |
RewardModelWithTies_seed_43 |
0.5706 | 0.4634 | 0 | 1.041 | 0.04882 | |
RewardModelWithTies_seed_44 |
0.5625 | 0.4565 | 0 | 1.042 | 0.04538 | |
seeds_mean |
0.5616 | 0.456 | 0.0007663 | 1 | 1.042 | 0.04309 |
seeds_std |
0.007696 | 0.006225 | 0.001084 | 0 | 0.0007847 | 0.005839 |
RewardModelWithTies_ensemble |
0.5695 | 0.4617 | 0 | 1.04 | 0.04309 |
68a17263085470cbd7d76413d2bcfe5f0d82b95ef801d1372152e30c40212a97The weights are a standard transformers sequence-classification checkpoint plus a small wrapper config, so load them through the thesis code, which restores the wrapper (max length, tie threshold, ...) exactly as it was trained:
git clone https://github.com/carlos-musetti/tesis
cd tesis/reward-model && uv sync
import sys; sys.path.insert(0, 'code')
from hub_upload import load_ensemble_from_hub, load_best_trial_from_hub
model = load_ensemble_from_hub('english_with_ties') # the final model
trial = load_best_trial_from_hub('english_with_ties') # the best single trial
reward = model.score(news_text, joke_text)
Set HF_HUB_NAMESPACE to the user or organisation that owns this repo if it is not your own account.
best_trial/model.safetensors 499 MBensemble/member_0/model.safetensors 499 MBensemble/member_1/model.safetensors 499 MBensemble/member_2/model.safetensors 499 MBFacebookAI/roberta-base