Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Latent-Foresight learns a latent world model end to end: an autoencoder compresses frozen DINOv2 features into a compact latent space, and a diffusion transformer (DiT), trained with a flow-matching (rectified flow) objective, predicts the latent of future frames. The autoencoder and the predictor are trained jointly, so the latent space is shaped to be predictable. Off-the-shelf DPT heads then decode the predicted features into semantic segmentation, depth and surface normals.

This repository contains the checkpoints trained on Cityscapes.

Paper

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Code

The official implementation can be found on GitHub: https://github.com/Sta8is/Latent-Foresight

Checkpoints

Sample Usage

Install the environment with uv and download the checkpoints:

git clone https://github.com/Sta8is/Latent-Foresight
cd Latent-Foresight
uv sync
source .venv/bin/activate
hf download Sta8is/Latent-Foresight_cs --local-dir checkpoints

For training and evaluation instructions, refer to the GitHub repository.

Citation

If you found Latent-Foresight useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!

@article{karypidis2026latent,
  title={Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models},
  author={Karypidis, Efstathios and Gidaris, Spyros and Komodakis, Nikos},
  journal={arXiv preprint arXiv:2610.01942},
  year={2026}
}

Acknowledgements

Our code is partially based on Dino-Foresight. We also thank the authors of DINOv2 and DPT for their work and open-source code.