Real-time, endless text-to-motion trained only on the CMU Motion Capture Database, so the weights can be used in commercial products.
Type a short English caption. The model keeps generating full-body motion forever, and you can switch the caption at any moment. A new caption takes effect at the next step (4 frames = 0.13 s), and the motion so far is kept as context, so the transition is continuous.
Author: やきますく (Yakimasuku) · Code and weights: Apache License 2.0
One continuous take (seed 1, not cherry-picked). The caption changes every 4 s: walk → Wave → run/jog → jump → Standing still.
walk |
run/jog |
jump |
Wave |
|---|---|---|---|
kick |
salsa dance |
boxing |
|
Seed 1, first take of each. The figure is a simple preview body drawn from the generated joint positions and rotations; it is not an SMPL mesh.
pip install -r requirements.txt
python live/live_server.py # then open http://localhost:8765
On Windows you can also double-click live/start_windows.bat.
python generate.py --sec 4 --out motion.npz "walk" "Wave" "run/jog" "jump"
Each caption is used for --sec seconds, in order, in one continuous motion. The output .npz is at 30 fps:
| key | shape | content |
|---|---|---|
positions |
(T, 22, 3) | world joint positions in meters, y up, SMPL joint order |
rotations |
(T, 22, 3, 3) | world joint rotations |
root_xz |
(T, 2) | pelvis path on the floor |
prompt |
(T,) | caption used for each frame |
parents |
(22,) | skeleton parents |
Values are rounded to 0.001 (1 mm for positions).
The model only knows the CMU motion descriptions. Short captions in the CMU style work best, for example:
walk, run/jog, jump, Wave, kick, salsa dance, boxing, Standing still, sit on high stool, stand up, walk backwards and turn, cartwheel, Mickey sneaking walk
The full list is in data/cmu_files.csv (column desc). Case doesn't matter: in our tests Run and run gave identical results, because the CLIP tokenizer lower-cases the text.
| Motion representation | 272 numbers per frame at 30 fps (root velocity and heading, joint positions, velocities and 6D rotations of 22 SMPL-order joints), following the 272-dim representation of MotionStreamer. CMU ASF/AMC is converted with our own converter (code/cmu_to272.py). |
| Tokenizer (7.5 M parameters) | Causal hierarchical VQ. Every 4 frames become 7 tokens: 1 global code (codebook 4096) plus 6 part codes (codebook 128 each) for torso, head, right arm, left arm, right leg and left leg. Encoder and decoder only look at the past, which allows streaming. |
| Generator (30.5 M parameters) | Decoder-only Transformer: 8 layers, width 512, 8 heads, RoPE. Context is 64 steps (8.5 s) and slides forever, which makes it endless. The text embedding (CLIP ViT-B/32) is added at every position, so the caption can be swapped mid-stream. Classifier-free guidance s = 4 (null text for 10% of training samples). |
Tokenizer reconstruction on the validation data (best step 18,000):
| part | position error (mm) | rotation error (°) |
|---|---|---|
| torso | 33.2 | 5.3 |
| head | 56.1 | 8.2 |
| right arm | 67.4 | 14.2 |
| left arm | 68.4 | 14.2 |
| right leg | 55.4 | 8.2 |
| left leg | 56.4 | 8.2 |
| all | 57.4 | 9.9 (global) |
Generator validation loss (cross-entropy per token) at the best step (9,500):
| global | torso | head | right arm | left arm | right leg | left leg | mean |
|---|---|---|---|---|---|---|---|
| 2.654 | 2.459 | 2.428 | 2.208 | 2.114 | 2.196 | 2.114 | 2.311 |
There is no standard text-to-motion benchmark for CMU-only models, so these numbers can't be compared directly with models trained on HumanML3D.
run alone runs for about 2 s and then slows to about 0.5 m/s. Use run/jog, which keeps running at about 2.2 m/s.Download allasfamc.zip from mocap.cs.cmu.edu and extract it. The folder contains subjects/.
Run:
python code/run_cmu.py --amc <extracted folder> --work <empty work folder>
This runs convert → tokenizer (2 h) → prep → generator (2 h). Checkpoints are written to <work>/ckpt as .pt files. To use them with the demo, convert them to weights/*.safetensors.
The released weights came from this pipeline. In the release copy, run_cmu.py was changed only to take the paths as arguments. That edited copy hasn't been run end to end again.
| path | content |
|---|---|
weights/generator.safetensors |
generator |
weights/tokenizer.safetensors |
causal tokenizer (encoder and decoder) |
weights/Mean.npy, weights/Std.npy |
normalization of the 272-dim features |
live/ |
real-time server and page |
generate.py |
generate to .npz |
code/ |
model code and the training pipeline |
data/cmu_files.csv |
CMU clip list with motion descriptions (used as captions) |
code/tvq.py, code/tvq272.py and code/vq_mix.py are shared modules from earlier research code, and their comments mention HumanML3D experiments. Only their model classes and helper functions are used here. No HumanML3D or AMASS data or weights were used to train this model.
LICENSE).data/cmu_files.csv is only the list of clip names and descriptions.Acknowledgment: The data used in this project was obtained from mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.
CMU のモーションキャプチャだけで学習した、文章から全身の動きを作るモデルです。止まらずに動きを作り続け、途中で文章を切り替えられます。商用でも使えます。
python live/live_server.py を実行して http://localhost:8765 を開くと、文章を入れてリアルタイムに動かせます(NVIDIA の GPU が必要です)。weights/generator.safetensors 122 MBweights/tokenizer.safetensors 35 MB