TQC humanoid, hand-edited to stop falling

An unofficial, hand-edited version of sb3/tqc-Humanoid-v3 from the RL Baselines3 Zoo. It wasn't retrained: a few of its weights were changed by hand.

The zoo's TQC humanoid is the fastest of its SAC, TD3 and TQC humanoids, but on current MuJoCo it falls in about two episodes out of three. This version runs the full episode 99.95% of the time and keeps most of the speed.

Made with WSP Lab, a tool that lets AI agents edit a model's weights directly, quickly and verifiably. This edit used its prototype. WSP Lab is coming soon: join the waitlist at wsp-lab.ai.

How long it took

• Total: 1 hour 58 minutes from downloading the zoo model to audited results, all on one laptop (Apple M5 Max) with no training. About 35 minutes of that went to waiting for a second editor, Claude Opus 5.5, which worked in parallel for comparison. Without it, the whole job takes about 1 hour 25 minutes.
• The first test was already most of the fix. Fable 5.1's first test, 9 minutes after it got the brief, cut falls from 63% to 5%.
• Budget: the edit used 30 test runs of about 10 seconds each. Most of the 27 minutes went to the agent's reasoning, not to compute.
• Timing source: every time above comes from the project's own file timestamps on 6 November 2026.

Results

2,000 episodes of Gymnasium's Humanoid-v4, on seeds 2,000,000 to 2,001,999, with deterministic actions:

Paired differences on the same seeds, with 95% bootstrap intervals:

• Against the unedited TQC: +2,538 (+2,430 to +2,645).
• Against SAC: +1,699 (+1,680 to +1,720). It scored higher in 1,999 of the 2,000 episodes.

The table was measured with the experiment's own float64 evaluation code. Loaded through Stable-Baselines3 as shown below, it scored 7,951 with 0.2% falls on a separate set of 500 seeds. Stable-Baselines3 computes in float32, which changes individual episodes but not the averages.

Videos

The same seed, side by side. The unedited model (left) falls after 5.7 seconds, and the edited one (right) runs the full 15 seconds. Seed 4 was picked, before any edit existed, as an episode where the unedited model falls.

The first four seeds, taken in order rather than chosen. The unedited model (top row) falls on three of them, and the edited model (bottom row) runs all four. On seed 3, where the unedited model stays up, it scores a little more, 8,067 against 7,950. The edit gives up a little top speed in exchange for not falling.

If the players don't load, the files are videos/side_by_side_seed4.mp4 and videos/first_four_seeds.mp4.

The videos were rendered with the experiment's own evaluation code. Running the downloaded file through Stable-Baselines3 can change individual episodes slightly, though not the averages.

What was changed

1. A repair for current MuJoCo. The zoo's Humanoid-v3 agents were trained when the 84 contact-force observations (indices 292 to 375) were always zero, because of a mujoco-py bug. Current MuJoCo fills them in, and the agents fall almost at once. The first-layer weights on those inputs are set to zero. The same repair takes sac-Humanoid-v3 from 447 to 6,291.
2. Perceived forward speed +0.2 m/s. The first-layer biases are shifted so the network reads its torso velocity, and every body's velocity, as 0.2 m/s faster than they really are. In response it runs a little slower and much more steadily.
3. Perceived sideways speed −0.15 m/s, shifted the same way, toward the side its torso faces.
4. Softer commands. The final layer's weights and biases are multiplied by 0.7, which scales the outputs before the tanh.

The edit changes 4,625 weights, and the repair zeroes another 21,504. The critics and the log-std head are the original's. Removing any one of the three edit parts makes it fall more often: 17.8%, 2.0% and 11.2% of episodes respectively, from 0.2%.

Usage

import gymnasium as gym
from huggingface_hub import hf_hub_download
from sb3_contrib import TQC

path = hf_hub_download(repo_id="erickvs/tqc-Humanoid-v4-hand-edited", filename="tqc-Humanoid-v4-edited.zip")
model = TQC.load(path)
env = gym.make("Humanoid-v4", render_mode="human")
obs, _ = env.reset(seed=0)
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, _ = env.step(action)
    if terminated or truncated:
        break

example.py scores the model on a range of seeds.

It was tested with stable-baselines3 2.9.2, sb3-contrib 2.9.2, gymnasium 1.4.0 and MuJoCo 4.0.0.

actor.safetensors holds just the six actor tensors, for use without Stable-Baselines3:

action = 0.4 * tanh(mu(relu(latent_pi.2(relu(latent_pi.0(obs))))))

Limits

• Environment: use Humanoid-v4, which has 376 observations. Humanoid-v5's observations are different.
• Deterministic actions only: sampling from the stochastic policy hasn't been tested.
• Fine-tuning: further training hasn't been tested. The zip has fresh optimizer states and no replay buffer.
• One base model: the edit was designed for this checkpoint, and the same values won't carry over to other agents.

How it was made

The model comes from an experiment in editing neural networks by hand, with no training. It used the prototype of WSP Lab, a tool for AI agents that's coming soon. Two AI editors worked independently under the same fixed rules: Claude Fable 5.1 and Claude Opus 5.5, both by Anthropic.

The rules banned gradients, optimizers, fitting weights to data and automated search. Each editor had 30 test runs on development seeds. Choosing between finished edits used separate selection seeds, and the final score used final seeds that neither editor could run.

This is Fable 5.1's edit. Opus 5.5 found the same idea on its own and scored 7,754, with 4.95% falls. An independent audit reproduced every score, rebuilt both edits from their scripts and tested each part by removal.

Credits and license

Derived from sb3/tqc-Humanoid-v3, from the RL Baselines3 Zoo by Antonin Raffin and contributors. It's released under the same MIT license, and the original copyright notice is in LICENSE. TQC comes from sb3-contrib.

This model isn't affiliated with or endorsed by the Stable-Baselines3 team.