StreamTalk official CFG checkpoints

Official checkpoints for StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring (ECCV 2026), a closed-loop method for real-time, long-horizon co-speech gesture generation.

• Paper · Hugging Face Paper page
• Project page · Code
• Generated inference data

These are hardware-agnostic FP32 generator state dictionaries for DiffusionDITNetPartsFixedExpressions2PostNormInteraction2. They were retrained on BEAT2 English with classifier-free condition dropout and selected with an accelerated H200 implementation of the StreamTalk generation procedure plus the official EMAGE/PantoMatrix AESK FGD metric.

The combined selection score is max(Speaker2 FGD / 0.383, All FGD / 0.293).

The checkpoints contain 407 CPU FP32 tensors and 71,167,501 parameters. Both load strictly into the originally released StreamTalk model schema. H200, BF16, CUDA Graphs, and four-GPU execution were training/evaluation infrastructure choices and are not encoded in the checkpoint files.

The reported values are accelerated EMAGE/PantoMatrix AESK FGD measurements, not paper-exact oracle measurements. A 1e-3 absolute FGD reporting tolerance was selected for engineering comparisons, but the final CFG=3 values have not been accepted against a direct B=1/M=1/full-window oracle. BC and DIV have not been re-evaluated for these retrained checkpoints.

The published inference NPZ bytes were independently re-scored with the released scorer (metric batch 16): Speaker 2 0.3788789702354345; all speakers 0.21764184426140076. These differ from the selection-time values by about 7e-15 and 2.97e-5, respectively.

Verify downloads before loading:

python tools/verify_pretrained.py --weights-dir checkpoints/pretrained

PyTorch checkpoints use pickle internally. Only load files downloaded from the linked StreamTalk release and matching the published SHA256 values.

End-to-end inference also requires WavLM Large, the bundled SimpleSpeechModel, SMPL-X neutral, and the BEAT2 retrieval database. Those runtime assets are independent of the generator checkpoint format.

Citation

@inproceedings{zhang2026streamtalk,
  title={StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring},
  author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Yang, Kaixing and Hoi, Steven},
  booktitle={European Conference on Computer Vision},
  year={2026},
  eprint={2608.01643},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.01643}
}