UnifoLM-ER-Flow

Project Page

UnifoLM-ER-Flow extends UnifoLM-ER-1 with interaction-centric world modeling and discrete action learning. It jointly aligns visual observations, language conditions, predicted future dynamic regions, and robot actions within a single vision-language model.

Dynamic Region Prediction

We use optical flow to extract dynamic regions that capture future scene changes, then train a VQ-VAE to encode them into fixed-length sequences of discrete tokens. Conditioned on the current image and a task description or action, the VLM directly predicts mask tokens for future dynamic regions, focusing on interaction subjects and the scene changes they induce to enable interaction-centric world modeling.

Demonstrations

Discrete Action Learning

We partition the unified action space into three components: end-effector (EEF) poses, end-effector joints, and lower-body joints. A separate residual vector quantization (RVQ) model is trained for each component to discretize action sequences. Building on UnifoLM-ER-1, we introduce discrete action tokens and mask tokens for future dynamic regions, jointly aligning visual, language, and action representations within a single VLM to obtain the UnifoLM-ER-Flow model.

Action Encoding Pipeline

Each continuous motion component is encoded independently with its own RVQ model. The resulting token sequences share the same timesteps and are synchronously fed into the VLM.

Unified Training Signals

Together, these signals connect spatial understanding, future-change prediction, and action representation in a unified multimodal model.