Qwen2.5-0.5B-Instruct PPO on GSM8K

这是 verl 的 Quickstart: PPO training on GSM8K dataset。

An experimental full-parameter PPO training run based on Qwen/Qwen2.5-0.5B-Instruct, trained with the verl framework on GSM8K.

This repository contains the merged Actor model in standard Hugging Face Transformers format. It does not contain the Critic, optimizer state, or the full distributed-training checkpoint.

Training

• Framework: verl
• Trainer path: V1, synchronous PPO (TaskRunnerV1 / PPOTrainerSync)
• Algorithm: PPO with GAE advantages and a rule-based GSM8K reward
• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Dataset: openai/gsm8k, 7,473 training examples and 1,319 test examples
• Training batch size: 256 prompts
• Epochs: 15
• Maximum prompt length: 512 tokens
• Maximum response length: 512 tokens
• Actor learning rate: 1e-6
• Critic learning rate: 1e-5
• Hardware: one NVIDIA RTX 4090D (24 GB)

The reward parser expects the final answer in the form #### number, matching the standard GSM8K rule implementation used by verl.

Evaluation

Evaluation uses 1,319 GSM8K test examples, one stochastic generation per prompt, temperature 1.0, top-p 1.0, and a maximum of 512 new tokens. Scores were computed with verl.utils.reward_score.gsm8k.compute_score.

The strict metric requires the #### number format. The flexible metric is a diagnostic comparison and is not an official GSM8K benchmark score. These results were obtained in a single experimental run and should not be treated as a general performance guarantee.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Palind/qwen25-0.5b-ppo-gsm8k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {
        "role": "user",
        "content": (
            "Solve the problem step by step and output the final answer "
            'after "####".\n\nWhat is 12 + 5?'
        ),
    }
]

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
answer = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print(answer)

Limitations

• This is a small research and learning artifact, not a production model.
• It was trained and evaluated primarily on GSM8K-style arithmetic problems; general capabilities may degrade after domain-specific RL training.
• Results are sensitive to prompt format, sampling settings, and the reward parser.
• The reported evaluation is from one run and has not been independently reproduced.

License

The base model is released under the Apache-2.0 license. Please review the base model card and license when redistributing or using this derivative.