PokerBench Qwen3-14B + LoRA + DPO v3 (multi-seat, sized)

DPO checkpoint addressing both major findings from the v1 DPO release:

1. Multi-seat training — v1 trained UTG-only and regressed at CO by 25.87 bb/100. v3 trains across all 6 seats (4686 preference pairs covering UTG/HJ/CO/BTN/SB/BB).
2. Built on v2 SFT (sized) — v1 SFT had bare "raise" labels with no sizing; v2 SFT was retrained with "raise N" labels recovering BB amounts from prompts. v3 DPO inherits the proper sizing.

This is the fourth LoRA layer stacked on Qwen/Qwen3-14B:

• Stage 1 SFT (paper-format full-data, EM 90.07%)
• Stage 2 SFT v2 (50/50 paper + production-PE-fixed, sized)
• Stage 3 DPO (multi-seat counterfactual EV preferences) ← this

Headline numbers — 4-stage evaluation

Stage 1 — Paper benchmark (regression check)

DPO doesn't break the SFT abilities — actually slightly improves preflop EM.

Stage 2 — PE 200 (production format)

Stage 3 — vs gpt-oss-120b (1000 hands head-to-head) ⭐

Beats the +3 bb/100 success threshold. This is the core RL goal — exploit non-GTO opponents.

Stage 4 — vs v2 SFT base (1000 hands self-play)

DPO v3 wins against its own SFT base by +20.25 bb/100 — proves the RL loop genuinely improves on SFT.

Per-seat breakdown (v3 vs v2 SFT)

The CO regression that motivated this v3 retrain is fully resolved (~80 BB swing).

Method (DPO recipe)

Self-play data collection (reused from v1)

• 5000 hands of 6-max NLHE: ckpt-mixed × 3 seats vs openai.gpt-oss-120b-1:0 × 3 seats on AWS Bedrock
• Team A (ckpt-mixed) +46.10 bb/100 baseline reproduced

Counterfactual EV preference extraction (new for v3)

• Multi-seat: extract pairs for ALL 6 hero seats per hand (v1 was UTG-only)
• For each seat's key hero decision: 2 candidate actions × N=10 MC rollouts × deepseek.v3.2 driving the rest of the table
• Resume support across runs to avoid recomputing
• Final dataset: 4686 pairs across positions:
  • SB: 1655 (most postflop spots)
  • UTG: 1271
  • CO: 1175
  • BTN: 393
  • HJ: 163
  • BB: 29
• EV gap median 2.00 BB, p75 4.36 BB, max 75.20 BB

DPO training

• TRL 1.5.1 DPOTrainer, ref_model=None (PEFT adapter-disable trick saves 28 GB)
• LoRA r=32, alpha=64, target=all-linear
• LR 5e-6, beta 0.1, sigmoid loss
• 147 steps × batch 32 = 1 epoch on 4686 pairs
• Wallclock: ~25 minutes

Final metrics:

• train_loss: 0.593
• rewards/margins: 0.514 (v1 was 0.19)
• rewards/accuracies: 73% (v1 was 64%)

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "ianlee1996/pokerbench-qwen3-14b-lora-dpo-v3")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-14B")

system_prompt = (
    "You are a specialist in playing 6-handed No Limit Texas Holdem. "
    "Output ONLY the optimal action with no explanation. "
    "Valid formats: 'fold', 'check', 'call', 'bet N', 'raise N', 'all-in'."
)
# Inference: temp 0.1, top-p 0.95, max_tokens 16

Honest reporting

• UTG seat showed -2.05 bb/100 vs the SFT base in Stage 4 — this is within statistical noise (1000 hands ≈ ±3 BB std). The same model is +19.5 bb/100 at UTG against gpt-oss-120b, so it's not a real regression. A v4 with higher UTG-data weighting is in development for users who want every seat strictly improved.
• HJ/BB had relatively few training pairs (163 / 29 respectively) because GTO play makes these seats fold preflop most of the time. The eval metrics for these seats are noisier than other seats.

Reproducibility

Pipeline + experiment configs at https://github.com/IanLiYi1996/PokerBench:

# 1. Self-play (~9 hours, ~$50 Bedrock)
.venv/bin/python -m scripts.collect_selfplay --config configs/eval/rl_selfplay_5k.yaml

# 2. Multi-seat counterfactual extraction (~14 hours, ~$30 Bedrock)
.venv/bin/python -m scripts.extract_preferences \
    --hand-logs data/rl/selfplay_5k.jsonl \
    --out data/rl/preferences_5k.jsonl \
    --adapter checkpoints/09_qwen3_14b_lora_mixed_v2_sized \
    --hero-seats all --n-mc 5 --max-workers 8 \
    --bedrock-model deepseek.v3.2

# 3. DPO (~25 min)
.venv/bin/python -m scripts.train --config configs/experiments/10_qwen3_14b_dpo_v3_on_v2_sft.yaml

The 4686 preference pairs and raw self-play logs are public at ianlee1996/pokerbench-rl-dpo.

Citation

@inproceedings{zhuang2025pokerbench,
  title={PokerBench: Training Large Language Models to become Professional Poker Players},
  author={Zhuang, Richard and Gupta, Akshat and Yang, Richard and Rahane, Aniket and Li, Zhengyu and Anumanchipalli, Gopala},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2025},
  url={https://arxiv.org/abs/2501.08328}
}

License

Apache-2.0, matching Qwen/Qwen3-14B and the PokerBench dataset.