qwen14b-code-trainer-v10-grpo

LoRA adapter for Qwen/Qwen2.5-Coder-14B-Instruct, reinforcement-learned with GRPO (DeepSeekMath) on a rule-based tool-call formatting reward. This is the V10 release — the first RL stage in the pipeline, trained on top of the V9 SFT adapter with a domain-adaptive pretraining (DAPT) merge.

Part of the Code-Trainer / RTPI pipeline (GitHub).

What changed from V9

1. Domain-adaptive pretraining (DAPT) — a LoRA adapter (qwen14b-dapt-offsec) trained on ~10K offensive-security code documents is merged into the base model before SFT, grounding the model in security tooling patterns.
2. GRPO reinforcement learning — a fresh LoRA is trained on top of the merged DAPT + V9 SFT weights using Group Relative Policy Optimization. The reward function targets tool-call formatting quality, not task completion — teaching the model to emit structurally correct <tool_call> tags with valid tool names and clean stop signals.
3. Adapter chain architecture — the final adapter sits on a 3-stage merged base: Qwen2.5-Coder-14B-Instruct → DAPT merge → V9 SFT merge → fresh LoRA (this adapter).
4. 12-tool Nexus schema — the reward function validates against the Nexus tool set: Read, Write, Edit, LS, Bash, Grep, Glob, WebFetch, TodoWrite, Skill, Task, ScopeCheck.

Adapter chain

The V10 GRPO adapter cannot be loaded directly on the base model — it expects the DAPT and V9 adapters to be merged first:

Qwen/Qwen2.5-Coder-14B-Instruct
  └─ merge: cmndcntrlcyber/qwen14b-dapt-offsec         (DAPT)
      └─ merge: cmndcntrlcyber/qwen14b-code-trainer-v9_mixed  (SFT)
          └─ LoRA: cmndcntrlcyber/qwen14b-code-trainer-v10-grpo (this adapter)

For deployment, all three are merged into the base and quantized to GGUF (see qwen14b-code-trainer-gguf).

Training data

• Prompt dataset: cmndcntrlcyber/code-trainer-v10-grpo-prompts
• Prompts: <1K curated prompts from three sources:

Each prompt is formatted via apply_chat_template with the Nexus system prompt and full tool definitions.

Reward function

Rule-based, 5-component weighted reward scoring each completion 0.0–1.0:

Training procedure

Training metrics

Logged every 10 steps. The reward signal is stable throughout training with no signs of reward hacking or distributional collapse:

Evaluation

Version comparison

Intended use

• Direct use: load the full adapter chain on top of Qwen/Qwen2.5-Coder-14B-Instruct for instruction-following code generation, tool calling, and multi-turn agent behaviour with improved <tool_call> tag formatting.
• Downstream: merge the full adapter chain into the base model and quantize to Q5_K_M GGUF for local serving via llama.cpp, Ollama, or LM Studio.
• Out of scope: this adapter was not trained for safety alignment, RLHF, or non-code tasks. The GRPO stage optimizes formatting, not factuality.

Deployment notes

• Recommended quant: Q5_K_M — preserves multi-token <tool_call> tag patterns better than Q4_K_M.
• Context length: 8,192 tokens recommended; trained at 768 max completion length but the base model supports 32K.
• DPO not applied: the pipeline's second RL stage (DPO on reasoning quality) was not executed for this release.

How to use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-Coder-14B-Instruct"
dapt_id = "cmndcntrlcyber/qwen14b-dapt-offsec"
sft_id = "cmndcntrlcyber/qwen14b-code-trainer-v9_mixed"
grpo_id = "cmndcntrlcyber/qwen14b-code-trainer-v10-grpo"

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(
    base_id, torch_dtype=torch.bfloat16, device_map="auto",
)

# Merge DAPT and SFT adapters into base weights
model = PeftModel.from_pretrained(model, dapt_id)
model = model.merge_and_unload()
model = PeftModel.from_pretrained(model, sft_id)
model = model.merge_and_unload()

# Load GRPO adapter (active LoRA)
model = PeftModel.from_pretrained(model, grpo_id)
model.eval()

messages = [
    {"role": "system", "content": "You are Nexus, a local-first coding agent with tool access."},
    {"role": "user", "content": "Read the file main.py and summarise its structure."},
]
inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True,
).to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Reproducibility

• Code: github.com/cmndcntrlcyber/code-trainer-pipeline
• Prompt dataset build:

  python -m src.phase4c_rl.data.build_grpo_prompts \
      --config src/config/pipeline-50.yml
  

• Training launch:

  python -m src.phase4c_rl.scripts.launch_grpo \
      --config src/config/pipeline-50.yml --wait
  

• W&B project: rtpi-phase4c-rl