TQC Ant, hand-edited: from 56% early endings to 2.6%

An unofficial, hand-edited version of sb3/tqc-Ant-v3 from the RL Baselines3 Zoo. It wasn't retrained. AI agents changed some of its weights directly.

On current MuJoCo, the zoo's TQC Ant ends 56% of its episodes early, and its score is half that of the zoo's TD3 Ant. This version ends only 2.6% early and scores twice as much. That puts it ahead of the zoo's SAC Ant and close behind its TD3 Ant.

Made with WSP Lab, a tool that lets AI agents edit a model's weights directly, quickly and verifiably. This edit used its prototype. WSP Lab is coming soon: join the waitlist at wsp-lab.ai.

Use Ant-v4 with contact forces on: gym.make("Ant-v4", use_contact_forces=True), which gives the 111 observations the zoo agents were trained with.

How long it took

• Total: 43 minutes from registering the downloaded model (20:25:51 on 6 November 2026) to an audited final score (21:08:36), all on one laptop (Apple M5 Max) with no training.
• What that includes: about 4 minutes of writing the agents' briefs, and about 1.5 minutes lost when the editing agent mis-copied its access token and had to be given it again.
• The release took 13 more minutes. A test download with no login scored 5,728, with no early endings, on 100 fresh seeds.
• Timing source: WSP Lab's event log.

Results

2,000 episodes of Gymnasium's Ant-v4 (contact forces on), on seeds 2,000,000 to 2,001,999, with deterministic actions:

Paired differences on the same seeds, with 95% bootstrap intervals:

• Against the unedited TQC: +2,832 (+2,743 to +2,919).
• Against SAC: +742 (+683 to +802).
• Against TD3: −108 (−145 to −71). TD3 is still ahead, because it runs faster.

A hidden failure it fixes too. Some episodes that don't end early still finish with the Ant stuck on its back, which the environment doesn't count as an early ending. On 300 fresh seeds, that happened in 30.5% of the unedited TQC Ant's surviving episodes, 1.4% of TD3's and 0.7% of this model's.

Robustness, on 500 seeds that weren't used while editing:

The table was measured with WSP Lab's float64 evaluation. Loaded through sb3-contrib as shown below, it scored 5,647 with 2.8% ended early on final seeds 2,000,000 to 2,000,499, against 5,663 in WSP Lab. The Ant's bounding gait is chaotic, so the float32 arithmetic Stable-Baselines3 uses can change single episodes completely, but the averages agree.

Videos

The second final seed, side by side, first 15 seconds. The unedited Ant (left) bounces out of the allowed height range at step 49. The edited one (right) keeps running. Seed 2,000,001 was the first final seed where the unedited model ended early inside 15 seconds.

Another final seed, with TD3 for comparison. The unedited Ant ends at step 180. The edited Ant and TD3 both keep going.

If the players don't load, the files are videos/original_vs_edited_seed_2000001.mp4 and videos/original_edited_td3_seed_2000004.mp4.

What was wrong, and what was changed

It doesn't fall over. It jumps out. An Ant-v4 episode ends when the torso leaves the height band from 0.2 to 1.0 m. The TQC Ant runs with a bounding gait, and after a hard landing it springs back up hard enough to carry its torso above 1.0 m. Nearly every early ending in its logs is that, with the torso still rising. With the height rule switched off, it runs the whole episode and scores about 4,100.

The edit changes the network's output layer, before the tanh:

1. Softer hip commands: hips 1 to 3 multiplied by 0.6 and hip 4 by 0.5. The hips power the bounce, and hip 4 was the most saturated output. Lower hip gains stop the over-bounce. In episodes that run the whole way, the edited Ant is also faster: 5.16 m/s against 4.09 for the unedited one.
2. Ankles 1 to 3 multiplied by 0.15. The unedited Ant holds these three ankles against their joint limits on 75 to 86% of its steps. It does that with a steady command that pushes into the limit, and it pays for that in the reward's control cost with no effect on movement. Scaling those commands down removes most of that cost and keeps the gait. Ankle 4 moves more and is left as it is.

A repair for current MuJoCo, included in this model: the zoo's v3 Ant agents were trained when the 84 contact-force observations were always zero, because of a mujoco-py bug. Current MuJoCo fills them in. As in the zoo's other v3 Ant and Humanoid agents, the first-layer weights on those inputs are set to zero. All the numbers in the tables, including the unedited and reference ones, use this repair.

Together that changes 23,303 of the actor's numbers: 21,504 zeroed by the repair, and 1,799 scaled by the edit. The critics and the log-std head are the original's. The edit program is in edit_program.json.

Usage

import gymnasium as gym
from huggingface_hub import hf_hub_download
from sb3_contrib import TQC

path = hf_hub_download(repo_id="erickvs/tqc-Ant-v4-hand-edited", filename="tqc-Ant-v4-hand-edited.zip")
model = TQC.load(path)
env = gym.make("Ant-v4", use_contact_forces=True, render_mode="human")
obs, _ = env.reset(seed=0)
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, _ = env.step(action)
    if terminated or truncated:
        break

example.py scores the model on a range of seeds. It was tested with stable-baselines3 2.9.2, sb3-contrib 2.9.2, gymnasium 1.4.0 and MuJoCo 4.0.0. actor.safetensors holds just the six actor tensors, for use without Stable-Baselines3:

action = tanh(mu(relu(latent_pi.2(relu(latent_pi.0(obs))))))

Limits

• Environment: tested on Ant-v4 with use_contact_forces=True only.
• Deterministic actions only: sampling from the stochastic policy hasn't been tested.
• Fine-tuning: further training hasn't been tested. The zip has fresh optimizer states and no replay buffer, and the critics were left as the original's.
• One base model: the edit was designed for this checkpoint, and the same values won't carry over to other agents.

How it was made

The model comes from an experiment that compared four ways for AI agents to investigate a model before editing it. In every arm a Claude Fable 5.1 agent made the edit, under the same fixed rules:

Every arm found that the hips drive the over-bounce. The extra gain in this model came from the ankle finding, which its editor made while editing.

The rules banned gradients, optimizers, fitting weights to data and automated search. Each editor had a fixed number of test runs on development seeds. Choosing between finished edits used separate selection seeds, and the final score used final seeds that no agent could run.

Credits and license

Derived from sb3/tqc-Ant-v3, from the RL Baselines3 Zoo by Antonin Raffin and contributors. It's released under the same MIT license, and the original copyright notice is in LICENSE. TQC comes from sb3-contrib.

This model isn't affiliated with or endorsed by the Stable-Baselines3 team.