Getting Over It, finished by a network trained in a JAX simulator

One small policy network that plays the real Getting Over It with Bennett Foddy from the bottom to the end of the game, in one continuous climb with no save/load. It was trained entirely in a JAX re-implementation of the game's physics and rules, then played the real game unchanged through a BepInEx plugin.

Code: krzim/getting-over-it-rl, branch jax-engine, a fork of miwate/getting-over-it-rl.

Results in the real game

Win rate 85%: 17 of 20 continuous climbs from the level start won, each allowed up to 5 minutes of play. The command and logs are in eval/.

Files

The policy

• Input: 105 floats, the plugin's observation:

  • the pot and hammer state;
  • 32 terrain ray distances;
  • route progress;
  • the last 4 actions;
  • a stall counter;
  • the gap to the best height so far.

  The exact layout is goi_env.client.OBS_FIELDS plus goi_jax.env.

• Output: a 2-D mouse movement in [-1, 1], multiplied by 12 before it is sent to the game. The game turns it into hammer motion.

• Network: an MLP with 256 and 256 tanh hidden units, a tanh-bounded mean and a fixed log std. There is a separate value network of the same shape.

• Playing the real game with it (from the code repo, with the game and plugin installed): python -m goi_jax.play policy.npz --backend real --no-ends --steps 9000.

• Computing an action from policy.npz with numpy alone:

import numpy as np
w = np.load("policy.npz")
x = np.clip((obs - w["obs_mean"]) / np.sqrt(w["obs_var"] + 1e-8), -10, 10)
h = np.tanh(np.tanh(x @ w["pi0_w"] + w["pi0_b"]) @ w["pi1_w"] + w["pi1_b"])
mean = np.tanh(h @ w["mu_w"] + w["mu_b"])
action = np.clip(mean + 0.5 * np.exp(w["log_std"]) * np.random.randn(2), -1, 1)   # play.py's default noise 0.5
mouse = action * 12.0

How it was trained

• Simulator. The game's physics is Unity 2020.3 Physics2D: Box2D 2.3.1 with Unity's revolute joint. It is ported to JAX, together with:

  • the player controller;
  • friction switching;
  • the space section's zero gravity, gravity wells, teleporters and win.

  A simulated step matches the game's own Box2D world bit for bit on recordings of the real game. It runs about 45,000-50,000 steps per second at 16,384 parallel games on one RTX 3090.

• PPO on the GPU. Rollouts, updates and the start curriculum all run on the device, PureJaxRL-style.

  • Episodes start from saved states along the whole route, drawn more often from sections the policy fails to cross.
  • Further starts come from a Go-Explore search in the simulator.
  • Reward is new route progress, as in the upstream project.

• Walls. A whole-route policy stalled at the Devil's Chimney (176 m), then at the garden furniture tower (289 m). Each time, a copy of it trained only from starts just below the wall, with more exploration noise, and passed it within about one GPU hour.

  • This network is the last copy: 2,512 updates, about 1.3 billion simulated steps over the whole lineage.
  • It kept the skills of the earlier copies and climbs the whole route alone.

The upstream project's networks and models were not used.