A LoRA adapter for
Qwen/Qwen2.5-0.5B-Instruct
trained with Group Relative Policy Optimization (GRPO) on GSM8K.
This is a research artifact, not a production math solver. Read the caveats below before using it.
Qwen/Qwen2.5-0.5B-Instructopenai/gsm8k, train split<think>...</think><answer>...</answer> template. The format reward was zero
for every training step.The adapter must be loaded on top of the base model with PEFT:
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_model = "Qwen/Qwen2.5-0.5B-Instruct"
adapter = "NeelContractor/grpo-qwen2.5-0.5b-gsm8k"
device = (
"cuda"
if torch.cuda.is_available()
else "mps"
if torch.backends.mps.is_available()
else "cpu"
)
dtype = torch.float16 if device in {"cuda", "mps"} else torch.float32
tokenizer = AutoTokenizer.from_pretrained(base_model)
if tokenizer.pad_token_id is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=dtype)
model = PeftModel.from_pretrained(model, adapter)
model.to(device).eval()
SYSTEM_PROMPT = (
"You are a careful math tutor. Solve the problem step by step.\n"
"Respond exactly in this format:\n"
"<think>\n"
"...your reasoning...\n"
"</think>\n"
"<answer>\n"
"...final numeric answer...\n"
"</answer>"
)
question = "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?"
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Problem: {question.strip()}"},
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
completion = tokenizer.decode(
output[0][inputs["input_ids"].shape[1] :],
skip_special_tokens=True,
)
print(completion)
The expected answer for the example is 72.
The adapter was trained with the prompt above, but it may return plain reasoning without the tags. Parse the final numeric answer defensively rather than assuming the tags are present.
| Setting | Value |
|---|---|
| Base model | Qwen/Qwen2.5-0.5B-Instruct |
| Adaptation | LoRA on all linear projections |
| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
| Dataset | openai/gsm8k, train, shuffled with seed 42 |
| Optimizer steps | 200 |
| Prompts per step | 1 |
| Generations per prompt | 8 |
| Max completion length | 512 tokens |
| Sampling temperature | 1.0 |
| Learning rate | 1e-5 |
| GRPO beta | 0.04 |
| Precision | FP16 |
| Hardware | 1 x Colab T4 |
| Training time | about 1h45m |
| Seed | 42 |
The reward was:
R = R_correct + 0.5 * R_format
R_correct is 1.0 for an exact numeric match with the GSM8K gold answer.
R_format is 1.0 only for exactly one non-empty <think> block followed by one
non-empty <answer> block with no extra text.
Training used the prompt shown above. Only the LoRA adapter was trained; the base model was frozen. No full-model merge was performed.
test splitseed=42, greedy decoding, max_new_tokens=512| Stage | Correct | Accuracy | Format adherence |
|---|---|---|---|
| Baseline (untrained) | 17/50 | 34.0% | 0.0% |
| GRPO adapter | 23/50 | 46.0% | 0.0% |
| Change | +6 | +12.0 pp | +0.0 pp |
Paired outcomes: 8 improved, 2 regressed, 15 both correct, 25 both wrong.
McNemar exact two-sided p = 0.109.
Lenient sensitivity check (gold value appears anywhere in the completion):
| Stage | Strict | Lenient upper bound |
|---|---|---|
| Baseline | 17/50 (34.0%) | 25/50 (50.0%) |
| GRPO adapter | 23/50 (46.0%) | 27/50 (54.0%) |
| Change | +12.0 pp | +4.0 pp |
adapter_config.json)NeelContractor. For issues or questions, open an issue in the project repository.
Qwen/Qwen2.5-0.5B-Instruct