Chair-task sit-down recogniser (Unitree G1)

A 224 K-parameter CNN that tells whether the person in front of the robot is standing, starting to sit down, or seated, from the robot's own chest camera. It exists to trigger one action: in the task "pull out the chair indicated by the human gesture, then push it back when the human starts sitting down", the robot needs to know when the sit-down begins.

No pose estimator and no motion-capture rig at inference — one small CNN on raw greyscale pixels. Runs at 680 FPS on a single CPU thread.

Usage

from models.posture_rgb160.modeling_posture import PostureRecognizer

rec = PostureRecognizer.from_pretrained(
    'DaoyuanZhu/wb_pointed_chair_pull_push_posture', device='cpu')
for bgr in camera_stream():                 # 640x480 BGR frames, in order, 30 FPS
    out = rec.push(bgr)
    if out['warmup']:                       # first ~1.1 s fills the ring buffer
        continue
    print(out['posture'], out['confidence'], out['language'])
    if out['posture'] == 'sitting_down' and out['confidence'] > 0.5:
        robot.push_chair_back()

push() is causal — only the current and past frames are used — so it is safe to run online. It keeps its own 33-frame ring buffer and samples 9 frames at stride 4 (a 1.07 s window).

RTMO native skeleton models

This package also contains two chest-camera models that use RTMO's native COCO17 2D skeleton rather than RGB pixels or a rasterised skeleton. Both use a causal 9-point window at stride 4, so inference keeps 33 frames of history.

RTMO itself returns all detected people as keypoints with shape (people, 17, 2) and scores with shape (people, 17). Select the target person before calling the recogniser. The example below uses the largest detection, which is suitable for a single dominant person; a multi-person deployment should replace that line with a person tracker.

import numpy as np
from rtmlib import RTMO
from models.posture_rtmo_skeleton.rtmo_skeleton_model import \
    ContinuousSkeletonRecognizer

rtmo = RTMO(
    onnx_model="/path/to/rtmo-m_16xb16-600e_body7-640x640-39e78cc4_20231211.onnx",
    model_input_size=(640, 640), backend="onnxruntime", device="cpu",
)
rec = ContinuousSkeletonRecognizer.from_pretrained(
    "DaoyuanZhu/wb_pointed_chair_pull_push_posture",
    variant="with_score",       # or "no_score"
    device="cpu",
)

for bgr in camera_stream():    # 640x480 BGR, every frame in order
    keypoints, scores = rtmo(bgr)
    if len(keypoints):
        person = int(np.argmax((scores > 0.3).sum(axis=1)))
        valid = float((scores[person] > 0.3).sum() >= 3)
        out = rec.push(keypoints[person], scores[person], valid)
    else:
        out = rec.push(np.zeros((17, 2), np.float32),
                       np.zeros(17, np.float32), 0.0)
    if not out["warmup"]:
        print(out["posture"], out["confidence"], out["probs"])

The with_score model keeps the per-keypoint confidence values. The no_score model ignores them, but both versions retain the frame-level valid flag so a missing RTMO detection is distinguishable from a real body at the image origin. The models return standing, sitting_down, or seated, plus probabilities and confidence.

Episode 78 RTMO demo

demos/episode_000078_rtmo_direct_score_compare/episode_000078_rtmo_direct_score_compare.mp4 is a 26.5-second rerun on the raw episode-78 chest video. The left side is the raw camera view; the right side is the skeleton drawn from a fresh RTMO pass; the bottom compares the with-score and no-score predictions. The original chest-camera source is 640x480 at 30 FPS; this side-by-side visualization is rendered at 1280x720. The companion JSON summary records the source video, RTMO checkpoint, frame range, and selected-person rate.

Exact image/data flow for the RTMO skeleton models

The episode-78 input frame passed to RTMO is the original chest-camera frame at 640x480 (BGR, 30 FPS). The video is not resized or cropped before the RTMO call. RTMO is configured with model_input_size=(640,640), so its own preprocessing scales the 640x480 frame to the model's 640x640 canvas while preserving the aspect ratio and padding the remaining area. RTMO then maps the detected COCO17 keypoints back to the original 640x480 image coordinates.

The posture classifier does not receive the 640x480 pixels, the 640x640 RTMO tensor, or the rendered 1280x720 demo image. It receives only the selected person's RTMO-native data: 17 (x,y) keypoints, the optional 17 keypoint scores, and the frame-level valid flag. Both models use a causal 9-frame window sampled with stride 4 (33 frames of causal history). The comparison video is 1280x720 only because it places the raw view, skeleton overlay, and probability text on one visualization canvas.

Raw MediaPipe coordinate wave model

models/wave_mediapipe_coordinates/ is a separate learned not_wave / wave model. It receives only fresh MediaPipe hand coordinates: two fixed hand slots × 21 landmarks × (x,y,z), plus two valid flags, for 128 values per frame. It does not receive RGB pixels, rasterized skeleton images, or confidence scores. Missing hands are zero-filled with valid=0; no temporal interpolation or previous-frame hold is used. The model uses the same causal 9 frames at stride 4 as the other temporal models.

from models.wave_mediapipe_coordinates.wave_coordinates import \
    MediaPipeWaveCoordinates

wave = MediaPipeWaveCoordinates.from_pretrained(
    "DaoyuanZhu/wb_pointed_chair_pull_push_posture", device="cuda")
out = wave.push(landmarks_2d_2x21x3, valid_2)
print(out["label"], out["wave_probability"])

For the released GPU demo environment, make the installed CUDA 13/cuDNN libraries visible to ONNX Runtime before constructing RTMO. The exact base directory is environment-specific; for the environment used here:

CUDA_PY_LIB=/home/dyz/miniconda3/envs/env_isaaclab/lib/python3.11/site-packages/nvidia
CUDA_LIBS=$(find "$CUDA_PY_LIB" -mindepth 2 -maxdepth 2 -type d -name lib -print | paste -sd: -)
export LD_LIBRARY_PATH="$CUDA_LIBS:/usr/local/cuda/targets/x86_64-linux/lib:${LD_LIBRARY_PATH:-}"

Use device="cuda" for the coordinate classifier, RTMO's backend="onnxruntime", device="cuda" for body keypoints, and MediaPipe's GPU delegate for fresh hand landmarks. Verify the RTMO log contains CUDAExecutionProvider and the MediaPipe log identifies the physical NVIDIA renderer; do not accept a CPU/llvmpipe fallback.

Episode 78 fresh dual-model demos

The two videos in demos/episode_000078_mediapipe_raw_coordinate_dual/ were regenerated from the original chest-camera video. Every 640x480 frame was sent to both fresh detectors: RTMO body pose and MediaPipe hands. RTMO's native coordinates go directly to the posture classifier; the same fresh MediaPipe coordinates go to both the existing gesture recognizer and the new wave classifier. No stored skeleton array is used.

The first video shows the new raw-coordinate wave output with the previous postprocessing (threshold=.60, gap fill up to 12 frames, minimum positive run 24 frames, standing gate). On episode 78 the raw model has 88 positive frames split by short flickers; the postprocessor merges them into one continuous 96-frame event (frames 100--195, zero-based). The second shows RTMO posture and the current MediaPipe gesture recognition together, with that continuous raw-coordinate wave event alongside them. CURRENT GESTURE is intentionally the instantaneous/event output of the existing gesture translator and may change between Open Palm, Wave, and Unknown; use the NEW RAW-COORD WAVE EVENT line for the continuous wave decision.

For readability only, the demo overlay linearly fills bounded gaps of up to 12 frames between two real MediaPipe hand detections. This does not alter the coordinates sent to the wave model: model input remains fresh, raw 640x480 MediaPipe data with zero-filled invalid slots. The displayed wave probability is also a causal EMA (alpha=.20); the event decision itself still uses the documented threshold/gap-fill/minimum-run postprocessing.

Both detector passes ran on an RTX 5080: RTMO via ONNX Runtime CUDA and MediaPipe via its GPU delegate. The 1280x720 size is display-only; detector input is still 640x480.

The same fresh pipeline was also run on episodes 62 and 63. Their outputs are in demos/episode_000062_mediapipe_raw_coordinate_dual/ and demos/episode_000063_mediapipe_raw_coordinate_dual/.

Runnable pure-skeleton dual pipeline

pipeline/run_skeleton_only.py is the complete per-frame pipeline for a new chest-camera video. It runs fresh RTMO body detection and fresh MediaPipe hand detection in the same loop, then feeds both skeleton streams to the learned models:

640x480 RGB frame
       ├── RTMO (GPU) ── COCO17 xy/scores ──┬── posture with-score (GPU)
       │                                    └── posture no-score (GPU)
       └── MediaPipe (GPU) ── hand xyz/valid ─── wave coordinates (GPU)

The RGB frame is required by RTMO and MediaPipe because they are the pose detectors. It is not an input to either classifier. Standing/sitting uses only RTMO coordinates, optional RTMO keypoint scores, and a valid flag; wave uses only MediaPipe's two hand slots, 21 normalized (x,y,z) landmarks per hand, and two valid flags. There is no RGB tensor, RGB crop, or rendered skeleton image in the classifier inputs. The overlay video is visualization only.

The uploaded checkpoints use the trained causal window of 9 sampled frames at stride 4 (33 input frames of history, about 1.07 s at 30 FPS). This is not a newly retrained “9 adjacent frames, stride 1” variant; changing to stride 1 requires a separate retraining run.

Run from the Hugging Face repository

git clone https://huggingface.co/DaoyuanZhu/wb_pointed_chair_pull_push_posture
cd wb_pointed_chair_pull_push_posture
python -m venv .venv
source .venv/bin/activate
pip install -r pipeline/requirements.txt

./pipeline/run_skeleton_only.sh \
  --input /path/to/chest_video.mp4 \
  --output-dir outputs/chest_video

GPU is the default: RTMO uses ONNX Runtime CUDA, MediaPipe uses its GPU delegate, and the PyTorch classifiers use CUDA. The script stops with a clear error instead of silently switching to CPU when CUDA is unavailable. Use --cpu explicitly for a CPU test. RTMO's ONNX file is downloaded by rtmlib on the first run; use --rtmo-model /path/to/model.onnx to provide it locally. The input can have another resolution; detector coordinates are mapped back to that resolution and then normalized to the posture model's 640x480 coordinate frame.

The command writes:

The wave event keeps the released post-processing: probability > 0.60, fill internal gaps up to 12 frames, require 24 positive frames, and gate the final event on standing. The displayed probability uses a causal EMA for readability; raw model input and final event logic remain unchanged. Use --no-video when only machine-readable outputs are needed.

Inputs and outputs

Model folder layout

Each model family now has its own folder containing its inference code, checkpoint files, and model-specific README:

For a clean standalone copy containing only the fresh RTMO + MediaPipe, pure-skeleton posture/wave pipeline, use skeleton_only_pipeline/. It includes its own runner, dependencies, model subfolders, hand task, and the regenerated 78/63 examples, so it can be used without mixing those files with the older RGB and 320-pixel experiments.

The root chair_inference.py remains the convenience entry point for the combined 320x240 RGB-modality pipeline.

Results

Held-out episodes (10 of 80; no episode appears in both train and test):

Sit-onset detection (probability of sitting_down + seated >= 0.5 held for 3 frames): 10/10 events detected, 0 missed, 0 false alarms, median 0.25 s before the labelled onset.

Per-episode frame agreement: mean 97.7 %, worst 93.8 %, best 99.5 %.

Speed

Batch size 1, one forward pass per camera frame.

At 30 FPS the per-frame budget is 33.3 ms, so a single CPU thread uses about 5 % of it. No GPU is needed.

How the labels were made

Training labels were not hand-annotated. They come from a synchronised markerless mocap recording of the same sessions, which is already resampled 1:1 onto the LeRobot episode timeline (episode e frame i is row i), so no timestamp alignment was needed.

From the 40-point 3D skeleton, normalised hip height h = (z_hip - z_floor) / (z_hip_standing - z_floor) is thresholded with hysteresis: h >= 0.95 standing, h <= 0.68 seated, and the sit-down window runs from the last standing-height frame to the start of the seated run. Using the hysteresis crossing rather than a velocity walk-back keeps the onset stable when the subject pauses part-way down. 79 of 80 episodes yield exactly one clean sit event (median 1.80 s); the remaining episode has none — the human genuinely stays standing.

label_posture.py in this repo reproduces the labels; train_rgb.py reproduces the model.

Two things worth knowing about the source data

The mocap pipeline's "primary" tracklet is the wrong person in 27 of 80 episodes — it locks onto the teleoperator or a bystander rather than the person who sits. The per-episode identity letters are not stable across episodes either. The subject is re-selected per episode by taking the tallest well-covered tracklet; the separation is unambiguous (seated subject 1.65-1.72 m, teleoperator ~1.49 m hunched in the VR rig, bystanders ~1.25 m). Using the primary tracklet directly would mislabel a third of the data.

The 2D pose landmarks shipped with the dataset are not usable for this. During labelled sitting_down frames cam_chest.valid averages 0.46, and 40 of 79 episodes are below 50 % coverage with several at zero; cam_head is 0.04 during the sit. The person is plainly visible in the video throughout — it is an identity-lock failure in the pose pass, not occlusion. That is why this model reads pixels instead, and why the keypoint baseline above misses 1 of 10 events.

Onset timing

The onset was tuned against the demonstrator's own reaction, measured as the robot's wrist-motion burst when pushing the chair:

End to end the trigger lands about 0.58 s before the demonstrator started pushing, which gives a controller headroom rather than chasing the event.

Limitations

• Deep forward lean can trigger it. If the person bends far over the table without lowering their hips, the model may call it sitting_down. In the worst held-out episode this fired 1.9 s early for that reason. The label calls it standing (hip height barely moves); the image genuinely looks like a sit beginning. If false triggers matter more than latency, require the probability to hold for ~10 frames instead of 3 (costs about 0.23 s).
• One scene, one subject. All 80 episodes are the same room, camera pose and person. Expect to fine-tune for a new environment.
• standing_up is not supported. Every episode ends while seated, so the class has no training examples. The label code emits it if the data contains it.
• Labels are machine-generated from mocap geometry, not human judgement. Boundary frames between classes are intrinsically fuzzy.

Files

Additional 320x240 multimodal models

The models/posture_rgb320/ directory contains the newer high-resolution models for RGB plus pose overlay, pure skeleton, and all three modalities fused together. See models/posture_rgb320/README.md for loading and inference examples.

This package includes all three checkpoints used by that runtime: rgb320_rgb_overlay_best.pt, rgb320_skeleton_best.pt, and rgb320_fusion_best.pt.

Frame-by-frame waving detector

The models/wave_mediapipe/ directory contains the MediaPipe hand-based waving detector used in the accompanying videos. It processes the complete video at the original resolution (640x480 for the supplied data), calls the hand landmarker on every frame, tracks both hands independently, and does not interpolate missing detections or hold old landmarks.

Install its dependencies and run it on a complete video:

pip install opencv-python mediapipe numpy
cd models/wave_mediapipe
python waving_detector.py --input input.mp4 --output output_waving.mp4

GPU delegate is used by default; add --cpu when MediaPipe GPU support is not available. The detector marks real hand landmarks and writes WAVING when either hand passes the open-palm plus lateral-motion rule. Full details and a Python API example are in models/wave_mediapipe/README.md.

Learned wave models

The models/wave_rgb320/ directory contains four learned binary classifiers for not_wave / wave. The recommended all_three model combines RGB, a pure hand-skeleton image, and an RGB image with the hand skeleton drawn on it. It is the learned wave model used in the combined posture/wave videos. See models/wave_rgb320/README.md for loading examples.

Relabeled chair-task annotations

annotations/push_chair_wave/ contains the complete 80-episode full_body_language_hf/push_chair re-labeling produced by the posture-fusion and wave all_three models. The combined labels use sitting_down/seated > wave > standing priority, with wave enabled only while the posture model predicts standing. Post-processing uses wave probability > 0.6, at least 24 consecutive frames, and fills gaps up to 12 frames (0.4 s). No image frames are interpolated. Every episode has one continuous wave span after this post-processing. See annotations/push_chair_wave/manifest.json for the exact schema and model sources.

Joint inference

For the same policy on a prepared episode, use chair_inference.py:

from chair_inference import predict_from_directory

out = predict_from_directory(
    "/path/to/aligned_modalities",
    device="cuda",              # use "cpu" if needed
)
print(out["semantic"])          # standing / wave / sitting_down / sitting
print(out["wave"]["probabilities"][:, 1])

The directory must contain five frame-aligned raster arrays: frames.npy, overlay.npy, skeleton.npy, hand_skeleton.npy, and hand_overlay.npy. They are uint8 arrays at (T,C,240,320). The posture model uses RGB + body overlay + body skeleton; the wave model uses RGB + hand skeleton + hand overlay. Both use nine causal frames at stride four. The script downloads the model files from models/posture_rgb320/ and models/wave_rgb320/ automatically and applies the same >0.6 / 24 frames / 12-frame gap post-processing as the uploaded annotations.

Reproducing the data pipeline

The complete source-to-HF workflow is documented in PIPELINE.md. It covers the historical Aria gaze/hand/SLAM and markerless synchronization provenance, LeRobot conversion, RTMO body and MediaPipe hand passes, markerless posture labels, 160/320-pixel modality preparation, posture/wave training, joint relabeling, and the dry-run-first HF upload command. The lightweight annotations/ directory is included for inspection; raw videos, parquet files, and multi-gigabyte NumPy training caches remain in the linked dataset repository.

Citation

Derived from DaoyuanZhu/wb_pointed_chair_pull_push_rgb.