Qwen3.5-9B · factorio-build · Evolve & Reinforce (round 1)
Writes one def build(world): program that builds a working iron smelting line in
factory-sim scenes, scored on unseen scenes by the
factorio-build
environment (FactorioGym). It keeps its general
planning ability, which the earlier checkpoints in this series lost.
The merged weights are at the top level; the GRPO LoRA adapter (rank 16) is in adapter/.
Evolve & Reinforce
- Evolve. LLM-guided program search produces builder programs that pass on training scenes.
- Distil in the model's own voice. Base Qwen3.5-9B is shown each evolved program and asked to explain the approach step by step and write the program itself. A reply is kept only if its program scores at least as well as the original: 279 of 800. Training pairs the prompt without the program with the model's own reply, so the targets stay close to what the model already writes.
- Light SFT (LoRA rank 32, 3 epochs, lr 1e-4) on those replies, plus 20% replay of the base model's own answers to general prompts: qwen3.5-9b-factorio-build-er-r1-sft.
- Reinforce. 30 GRPO steps with prime-rl (LoRA rank 16, group size 16, 128 rollouts per step, temperature 0.7). Train success went from 23% to 86%.
Prompt version v2 of factorio-build (reason step by step, then the program), thinking off.
Results
Factorio success: 8 scenes per row, 16 rows × 4 samples, temperature 0.6, thinking off. PlanBench Blocksworld (500 instances each, thinking off, temperature 0) measures general planning.
| Model | Factorio val | Factorio holdout | PlanBench plan generation | PlanBench plan execution | Reply tokens (plan gen.) |
|---|---|---|---|---|---|
| Qwen3.5-9B (base) | 2.3% | 1.6% | 53.2% | 39.8% | 1,975 |
| SFT on 800 bare programs + 20 GRPO steps (earlier recipe) | 85.9% | 84.0% | 2.8% | 3.0% | 64 |
| Own-voice SFT (round 1 warm start) | 22.3% | – | 49% (first 100) | – | 1,859 |
| Own-voice SFT + 30 GRPO steps (this model) | 85.6% | 71.9% | 46.4% | 27.8% | 1,913 |
The earlier recipe fine-tuned on another model's bare programs, and the model stopped reasoning before answering (64-token replies) and lost general planning. This recipe keeps the base model's reasoning and most of its planning ability, at a lower holdout score. Plan execution still drops by 12 points, which the next round targets.
Use
messages = [{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}]
# chat_template_kwargs={"enable_thinking": False}, temperature=0.6
The prompts come from the factorio-build environment (prompt_version="v2").