SynthPose (MMPose HRNet48+DarkPose variant)

The SynthPose model was proposed in OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics by Yoni Gozlan, Antoine Falisse, Scott Uhlrich, Anthony Gatti, Michael Black, Akshay Chaudhari.

Intended use cases

This model uses DarkPose with an HRNet backbone. SynthPose is a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data.
More details are available in OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics.
This particular variant was finetuned on a set of keypoints usually found on motion capture setups, and include coco keypoints as well.

The model predicts the following 52 markers:

[
    'nose',
    'left_eye',
    'right_eye',
    'left_ear',
    'right_ear',
    'left_shoulder',
    'right_shoulder',
    'left_elbow',
    'right_elbow',
    'left_wrist',
    'right_wrist',
    'left_hip',
    'right_hip',
    'left_knee',
    'right_knee',
    'left_ankle',
    'right_ankle',
    'sternum',
    'rshoulder',
    'lshoulder',
    'r_lelbow',
    'l_lelbow',
    'r_melbow',
    'l_melbow',
    'r_lwrist',
    'l_lwrist',
    'r_mwrist',
    'l_mwrist',
    'r_ASIS',
    'l_ASIS',
    'r_PSIS',
    'l_PSIS',
    'r_knee',
    'l_knee',
    'r_mknee',
    'l_mknee',
    'r_ankle',
    'l_ankle',
    'r_mankle',
    'l_mankle',
    'r_5meta',
    'l_5meta',
    'r_toe',
    'l_toe',
    'r_big_toe',
    'l_big_toe',
    'l_calc',
    'r_calc',
    'C7',
    'L2',
    'T11',
    'T6',
]

Where the first 17 keypoints are the COCO keypoints, and the next 35 are anatomical markers.

Usage

Installation

This implementation is based on MMPose. MMpose requires torch, and the installation process is the following:

pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.1"
mim install "mmdet>=3.1.0"
mim install "mmpose>=1.1.0"

Image inference

Here's how to load the model and run inference on an image:

from huggingface_hub import snapshot_download
from mmpose.apis import MMPoseInferencer

snapshot_download(repo_id="yonigozlan/synthpose-hrnet-48-mmpose", local_dir="./synthpose-hrnet-48-mmpose")
inferencer = MMPoseInferencer(
    pose2d='./synthpose-hrnet-48-mmpose/td-hm_hrnet-w48_dark-8xb32-210e_synthpose_inference.py',
    pose2d_weights='./synthpose-hrnet-48-mmpose/hrnet-w48_dark.pth'
)

url = "http://farm4.staticflickr.com/3300/3416216247_f9c6dfc939_z.jpg"
result_generator = inferencer([url], pred_out_dir='predictions', vis_out_dir='visualizations')
results = next(result_generator)

The following visualization will be saved:

Where the keypoints part of the skeleton are the COCO keypoints, and the pink ones the anatomical markers.

Video inference

To run inference on a video, simply replace the last two lines with

result_generator = inferencer("football.mp4", pred_out_dir='predictions', vis_out_dir='visualizations')
results = [result for result in result_generator]

Training

Finetuning a model using SynthPose can be done by adapting the td-hm_hrnet-w48_dark-8xb32-210e_merge_bedlam_infinity_coco_3DPW_eval_rich-384x288_pretrained.py config on the following MMPose fork.
To create annotations on a synthetic dataset (such as BEDLAM) using SynthPose, the tools present in this repository can be used (better documentation to come).