SAC hopper, hand-edited to stop falling

An unofficial, hand-edited version of sb3/sac-Hopper-v3 from the RL Baselines3 Zoo. It wasn't retrained. AI agents changed a few of its weights directly.

On current MuJoCo, the zoo's SAC hopper falls in 98% of episodes, almost always around step 300. This version didn't fall in any of 2,000 test episodes, and it scores higher than the zoo's trained SAC, TD3 and TQC hoppers.

Made with WSP Lab, a tool that lets AI agents edit a model's weights directly, quickly and verifiably. This edit used its prototype. WSP Lab is coming soon: join the waitlist at wsp-lab.ai.

How long it took

• Total: 36 minutes from registering the downloaded model to an audited final score (19:17:52 to 19:54:16 on 6 November 2026), all on one laptop (Apple M5 Max) with no training. That includes 4 minutes of writing the agents' briefs. The release took 6 more minutes.
• The fix itself was found much sooner. One of twelve Claude Haiku 5.5 agents, investigating the same model in a separate arm of the experiment, ran a one-number version of the fix 37 seconds after starting. It scores 3,575 with no falls on the final seeds.
• Timing source: every time above comes from WSP Lab's event log and the release files' timestamps.

Results

2,000 episodes of Gymnasium's Hopper-v4, on seeds 2,000,000 to 2,001,999, with deterministic actions:

Paired differences on the same seeds, with 95% bootstrap intervals:

• Against the unedited SAC: +2,483 (+2,467 to +2,497).
• Against TD3: +119 (+118 to +119). It scored higher in all 2,000 episodes.

Robustness, on 500 seeds that weren't used while editing:

The table was measured with the experiment's own float64 evaluation code. Loaded through Stable-Baselines3 as shown below, it scored 3,654.0 with no falls on final seeds 2,000,000 to 2,000,499. No single episode differed by more than 7.4 points from the experiment's own evaluation.

Videos

The same seed, side by side. The unedited model (left) falls at step 304, 2.4 seconds in, and the edited one (right) hops the full 8 seconds. Seed 2,000,000 is simply the first final seed. It wasn't chosen.

The next seed, with TD3 for comparison. The unedited model falls at step 302. The edited model and TD3 both finish, and the edited one is a little faster.

If the players don't load, the files are videos/original_vs_edited_seed_2000000.mp4 and videos/original_edited_td3_seed_2000001.mp4.

What was wrong, and what was changed

The failure: the unedited hopper speeds up hop by hop, from about 2.3 m/s at step 100 to 3 m/s at step 200 and 4.5 to 5 m/s by step 300. Its thigh swings further back on each hop and the hops get lower, until the torso drops below the height limit. With no start noise at all, every episode ends at exactly step 296. The rare episodes that survive cruise at about 2.55 m/s. The network uses its sense of its own speed to set its pace, and it reads that speed too low.

The edit makes it perceive more speed than it has, so its own pacing holds back:

1. Perceived forward speed +0.3 m/s, in the 23 first-layer units that listen most to forward speed. Each of their biases is raised by 0.3 times its weight on the forward-speed input. That's exactly what those units would see if the torso were going 0.3 m/s faster.
2. A leg offset that switches on with speed. Three units the network never used are rewired. One reads forward speed, and two turn it into a ramp that shifts the leg command by −0.6 (before the tanh) once the hopper moves at 1 m/s or more, and not at all while it starts up. It makes the gait slightly faster. Applied all the time, the same offset made the hopper tip over at the start under heavy start noise.

The edit changes 809 of the actor's 69,635 numbers. The critics and the log-std head are the original's. The full edit program is in edit_program.json.

The one-line version. Most of the gain comes from the speed shift alone, applied to every first-layer unit. On the unedited model it is:

# actor.latent_pi.0 is the first layer; observation 5 is the torso's forward velocity
sd["actor.latent_pi.0.bias"] += 0.15 * sd["actor.latent_pi.0.weight"][:, 5]

That scores 3,595 with no falls on the same 2,000 final seeds, still above TD3.

Usage

import gymnasium as gym
from huggingface_hub import hf_hub_download
from stable_baselines3 import SAC

path = hf_hub_download(repo_id="erickvs/sac-Hopper-v4-hand-edited", filename="sac-Hopper-v4-hand-edited.zip")
model = SAC.load(path)
env = gym.make("Hopper-v4", render_mode="human")
obs, _ = env.reset(seed=0)
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, _ = env.step(action)
    if terminated or truncated:
        break

example.py scores the model on a range of seeds.

It was tested with stable-baselines3 2.9.2, gymnasium 1.4.0 and MuJoCo 4.0.0.

actor.safetensors holds just the six actor tensors, for use without Stable-Baselines3:

action = tanh(mu(relu(latent_pi.2(relu(latent_pi.0(obs))))))

Limits

• Environment: tested on Hopper-v4 only.
• Deterministic actions only: sampling from the stochastic policy hasn't been tested.
• Fine-tuning: further training hasn't been tested. The zip has fresh optimizer states and no replay buffer, and the critics were left as the original's.
• One base model: the edit was designed for this checkpoint, and the same values won't carry over to other agents.

How it was made

The model comes from an experiment comparing ways for AI agents to investigate a model before editing it. In each of four arms, a Claude Fable 5.1 editor made the edit, and every arm worked under the same fixed rules:

All four arms found the same cause, and all four edits beat the zoo's trained SAC, TD3 and TQC hoppers.

The rules banned gradients, optimizers, fitting weights to data and automated search. Each editor had a fixed number of test runs on development seeds. Choosing between finished edits used separate selection seeds. The final score used final seeds that no agent could run. An auditor reran the investigators' claims on fresh seeds, and 93% to 100% of the ones that could be rerun held up, depending on the arm.

Credits and license

Derived from sb3/sac-Hopper-v3, from the RL Baselines3 Zoo by Antonin Raffin and contributors. It's released under the same MIT license, and the original copyright notice is in LICENSE. SAC comes from Stable-Baselines3.

This model isn't affiliated with or endorsed by the Stable-Baselines3 team.