🖥️ GitHub | 🤗 Hugging Face | 🎮 Demo (Space) | 📑 Blog | 🎬 Video | 🔴 24/7 Live
MESA (Multi Entity Simulation Architecture) is a town simulation in which many residents live and walk around a procedurally generated city. The MESA town is streamed live on YouTube 24 hours a day (watch the live stream), and this project builds world models on top of that always-running world.
This repository contains a small action-conditioned world model that has learned what a MESA resident sees when they walk.
Given the last few first-person frames and an action (forward / turn left / turn right / backward), the model predicts what the resident will see next. It does this in the feature space of a frozen DINOv2 encoder, following the DINO-WM recipe, and a separate decoder turns the predicted features back into images so you can watch the model "dream".
MESA is not just a dataset generator. It is a living town that runs around the clock and is broadcast live on YouTube 24/7. Residents walk the streets, talk to each other, make friends and go about their day in real time. Viewers can even move into the town from the YouTube chat (!join).
This world model is built on that world:
The MESA World Model Simulator Space loads the weights from this repository. Press forward / left / right / back and the Space renders two videos side by side:
Untick "re-imagine from what is actually visible each time" to keep dreaming on top of the model's own predictions, and you can watch the errors build up over time.
first-person image ──DINOv2 ViT-S/14 (frozen)──> z_t (16×16 patch tokens × 384) ─┐
├─ World model (Transformer) ─> z_{t+1}
action a_t ∈ {forward, left, right, backward} ─────────────────────────────────────┘
(z_t, z_{t+1}) ── Inverse dynamics ──> a_t
z ── Decoder ──> 128×128 image (only used to look at the imagination, never for planning)
| Component | Architecture | Params | Notes |
|---|---|---|---|
| Encoder | DINOv2 ViT-S/14 (facebookresearch/dinov2:dinov2_vits14) |
21M (frozen) | 224×224 image → 16×16 patch tokens, 384-d each. Not stored here; loaded from torch.hub |
| World model (forward dynamics) | Pre-norm Transformer encoder, width 384, 6 layers, 8 heads | 11.0M | Context of 3 frames (768 tokens). Action embedding is added to every token. A frame-level causal mask lets frame i attend only to frames 0..i. Predicts a residual on top of the previous frame's features, so training starts from "nothing changes" |
| Decoder | Conv up-sampler (16×16 → 128×128) with residual blocks | 4.9M | Also receives the last real frame as a context image, so wall textures and colors can be copied from it. Trained with L1 + VGG16 perceptual + PatchGAN losses and EMA weights |
| Inverse dynamics | Linear reduction (384 → 32) + 3-layer MLP | 8.7M | (z_t, z_{t+1}) → logits over the 4 actions |
Each step either moves the resident 0.2 cells or rotates them by 12°.
| Index | Action | config.json label |
|---|---|---|
| 0 | forward | 前進 |
| 1 | turn left | 左 |
| 2 | turn right | 右 |
| 3 | backward | 後退 |
Walking into a building or tree is blocked, and the view stays the same. The model has learned to predict this as well.
| File | Contents |
|---|---|
world_model.safetensors |
World model (forward dynamics) |
decoder.safetensors |
Features → thumb×thumb image decoder |
inverse_dynamics.safetensors |
(current, next) features → action |
feat_stats.safetensors |
Mean mu and std sd used to standardize DINOv2 features (required before feeding the world model) |
config.json |
Model sizes, action order, renderer settings, the town seeds used for training and evaluation, and the full training config (train.preset) |
The loader, renderer and simulator code live in the mesa_wm package of the Space.
pip install torch numpy pillow safetensors huggingface_hub
hf download oggata/mesa-world-model-sim --repo-type space --local-dir mesa-world-model-sim
cd mesa-world-model-sim
from huggingface_hub import snapshot_download
from mesa_wm import Simulator, save_gif
sim = Simulator(snapshot_download("oggata/mesa-world-model-sharp-v01"), device="cpu") # or "cuda"
# Stand on a road in a town that was never used for training
st = sim.reset(seed=sim.cfg["maps"]["test_seeds"][0], start_seed=0)
# Walk forward 6 steps, then turn left 6 steps
real, imagined, st = sim.run(st, [0] * 6 + [1] * 6)
save_gif(real, imagined, "walk.gif") # left: real renderer, right: imagination
sim.run(..., sync=False) keeps imagining from the previous imagination instead of re-encoding what is actually visible.
To run the Gradio app locally:
MODEL_REPO=oggata/mesa-world-model-sharp-v01 python app.py
All data is generated on the fly, so no external dataset is needed.
Everything is trained from scratch in a single notebook (colab/world_model_mesa.ipynb). The notebook has three presets. The weights in this repository use sharp.
| Preset | Tokens | Decoded image | World model (width · layers · params) | WM steps | Rollout | Decoder | A100 time |
|---|---|---|---|---|---|---|---|
fast |
8×8 | 64×64 | 256 · 6 · ~5M | 12k | 1 | L1 only | ~10 min |
sharp ✅ |
16×16 | 128×128 | 384 · 6 · ~11M | 20k | 2 | L1 + perceptual + GAN, context image | 40–60 min |
sharper |
16×16 | 128×128 | 512 · 8 · ~26M | 30k | 2 | wider, 2 res blocks | 1.5–2.5 h |
Key hyper-parameters for sharp: batch 32, AdamW with lr 3e-4. The world model is trained with a 2-step rollout loss, feeding its own prediction back as input, which reduces blurring over long horizons. The decoder is trained for 10k steps with perceptual weight 0.2 and GAN weight 0.1. To make it robust to imagined features, training adds noise of 0.3 to the features and uses imagined features for 25% of samples. The inverse-dynamics head is trained for 3k steps.
The model is evaluated on towns never seen during training. It gets the last 3 real frames plus the next 15 actions, and its imagined features are compared with the features from actually walking those actions. The baseline predicts "nothing changes", reusing the last observed frame.
Reference numbers from the fast preset (book, chapter 14). Feature MSE, lower is better:
| Horizon | World model | "Nothing changes" baseline |
|---|---|---|
| 1 step | ≈ 0.22 | ≈ 0.49 |
| 15 steps | ≈ 1.0 | ≈ 1.54 |
The world model beats the baseline at every horizon. The sharp model in this repository uses 4× more tokens, a larger model and a rollout loss. Its own numbers are printed by the notebook but have not been reported in this card yet.
The notebook also evaluates:
fast), so fine details such as text on shop signs are not reproduced.sharp weights were trained without other residents (NPCs) (NPCS = 0). The architecture supports NPCs as extra tokens, which pass the residents' positions as a tensor, but these weights do not use them.config.json stores action labels in Japanese. Use the integer indices above.This is a research and education model. It is meant for learning how latent world models work (forward and inverse dynamics, imagination rollouts, planning in latent space) on a small, fully controllable world, and as a building block for experiments with agents in the MESA town simulation.
If you find this work useful, please cite this repository and the works it builds on:
@misc{mesa_world_model_2026,
title = {MESA World Model: an action-conditioned DINO-WM style world model for the MESA town simulation},
author = {oggata},
year = {2026},
howpublished = {\url{https://huggingface.co/oggata/mesa-world-model-sharp-v01}}
}
@article{zhou2024dinowm,
title = {DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning},
author = {Zhou, Gaoyue and Pan, Hengkai and LeCun, Yann and Pinto, Lerrel},
journal = {arXiv preprint arXiv:2411.04983},
year = {2024}
}
@article{oquab2023dinov2,
title = {DINOv2: Learning Robust Visual Features without Supervision},
author = {Oquab, Maxime and Darcet, Timoth{\'e}e and Moutakanni, Th{\'e}o and others},
journal = {arXiv preprint arXiv:2304.07193},
year = {2023}
}
The weights in this repository are released under the Apache 2.0 License. The frozen DINOv2 encoder is downloaded separately from facebookresearch/dinov2 and is subject to its own license.
Thanks to the authors of DINO-WM for the world-model recipe, DINOv2 for the visual features, and Hugging Face for hosting the model and the Space.
world_model.safetensors 45 MBinverse_dynamics.safetensors 35 MBdecoder.safetensors 32 MBfeat_stats.safetensors 3.2 kB