RL-Struct: Bridging the Structure Gap

中文版本 | 📚 Paper

We introduce RL-Struct, a lightweight Reinforcement Learning framework designed to solve the "Structure Gap"—the tension between probabilistic token generation and deterministic structured formats (e.g., JSON). By leveraging GRPO (Gradient Regularized Policy Optimization) and a Multi-dimensional Reward Function, our model achieves superior structural reliability without the high inference latency of constrained decoding.

🚀 Key Features

• Multi-dimensional Reward Function: Decomposes the objective into Structure, Format, Validity, Correctness, and Length.
• Efficient Training: Uses GRPO to eliminate the critic network, reducing VRAM usage by ~40% compared to PPO.
• Emergent Curriculum: The model spontaneously learns syntax (how to speak) before semantics (what to say).
• High Performance: Achieves 89.7% Structural Accuracy and 92.1% JSON Validity on complex recipe generation, outperforming LLaMA-3-8B and GPT-3.5.

📊 Model Details

• Base Model: Qwen/Qwen3-4B-Instruct-2507
• Training Method: GRPO (Reinforcement Learning) + LoRA
• Task: Structured Output Generation (JSON Recipes, GSM8K-JSON, ToolUse)
• License: Apache-2.0

🛠️ Usage

The following is the system prompt:

You are a precise recipe assistant. Always respond in the following JSON format:
{
  "reasoning": "Your step-by-step reasoning here...",
  "answer": "{\"name\": \"Recipe Name\", \"nutrition\": \"Calories: ..., Protein: ..., Fat: ...\"}"
}
Do not include any other text, explanations, or markdown. Only output valid JSON.

📈 Performance