R3D-18 for UCF-101 Action Recognition

Fine-tuned 3D ResNet-18 for video action recognition, trained as part of the video pipeline in human-action-classification -- a framework that also ships a pose/scene-based image pipeline (MediaPipe + timm) alongside this 3D-CNN video pipeline. Lightweight and fast; a good baseline and transfer-learning starting point rather than a top-accuracy model -- see MC3-18 in the Model Zoo below for the best-performing model in this project's UCF-101 lineup.

---

Performance

These are the values the training script's own validation loop saved into the checkpoint at its best epoch, not a separate benchmark run -- see Evaluation Protocol below for exactly how they were produced.

---

Evaluation Protocol

Metrics above come from VideoTrainer.validate() in hac.video.training.train, run on UCF-101 split 1's official test list (3,783 videos), at the checkpoint's best-performing epoch. Each clip: 16 frames sampled uniformly across the full video, resized preserving aspect ratio to roughly 128x171, center-cropped to 112x112, normalized with Kinetics-400 statistics -- a single center clip per video, no test-time augmentation or multi-crop averaging.

---

UCF-101 Model Zoo

Models from this project trained on UCF-101 split 1 with the same hac.video.training.train pipeline, for direct comparability:

---

Usage

Install Dependencies

Not yet published on PyPI -- install from source:

git clone https://github.com/dronefreak/human-action-classification
cd human-action-classification
pip install -e .

Load the Model from Hugging Face

import json
import torch
from huggingface_hub import hf_hub_download
from hac.video.models.classifier import Video3DCNN

config_path = hf_hub_download(repo_id="dronefreak/r3d-18-ucf101", filename="config.json")
weights_path = hf_hub_download(
    repo_id="dronefreak/r3d-18-ucf101",
    filename="r3d18-ufc101-split-1.pth",
)

with open(config_path) as f:
    config = json.load(f)

model = Video3DCNN(
    num_classes=config["num_classes"],
    model_name=config["model_type"],
    pretrained=False,
)

checkpoint = torch.load(weights_path, map_location="cpu", weights_only=False)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

Run Inference on a Video

The repo's VideoPredictor wraps frame sampling, transforms, and the forward pass end-to-end:

from hac.video.inference.predictor import VideoPredictor

predictor = VideoPredictor(model_path=model_path, device="cpu")
result = predictor.predict_video("path/to/video.mp4", top_k=5)

print(result["top_class"], result["top_confidence"])
for pred in result["predictions"]:
    print(f"  {pred['class']}: {pred['confidence']:.3f}")

---

Training Configuration

Rows marked "checkpoint ..." are read directly out of the optimizer/scheduler state and config dict stored inside r3d18-ufc101-split-1.pth. Rows marked "training script default" reflect hac.video.training.train's CLI defaults at the time of training but weren't independently confirmed for this exact run -- no separate run-config file was saved alongside the checkpoint.

---

Use Cases

✅ Best for:

• Baseline comparisons
• Transfer learning starting point
• Educational purposes
• Fast prototyping

⚠️ Consider alternatives for:

• Maximum accuracy on UCF-101 -- use MC3-18 (87.05%)
• Real-time / single-frame inference -- use one of the project's spatial (2D-CNN) models instead of a clip-based 3D-CNN

---

Known Limitations

• Fixed to UCF-101's 101 action classes -- no open-vocabulary or unseen-action support.
• Whole-clip classification only: no temporal localization, so a video containing multiple actions gets a single label.
• 16-frame uniform sampling can miss brief or rapid actions embedded in a longer clip.
• Precision (84.2%) noticeably exceeds recall (83.1%) on the test set -- the model is more prone to missing/confusing actions than to over-confident false positives.
• Trained only on UCF-101's largely trimmed, single-action YouTube clips; performance on untrimmed, multi-person, or surveillance-style footage is unverified.

---

Repository Contents

r3d18-ufc101-split-1.pth
config.json
demo.gif
README.md

config.json doubles as the Hub's download-count query file: since this repo has no library_name integration the Hub recognizes, it falls back to counting requests against config.json (per Hugging Face's download-stats docs) -- the loading snippet above fetches it as part of normal usage, so downloads register.

---

Related Resources

• MC3-18 for UCF-101 -- best-accuracy sibling model in this project's UCF-101 model zoo
• human-action-classification -- the training/inference framework used to produce this checkpoint

---

Citation

If you use this model, please consider citing the UCF-101 dataset, the R3D architecture, and the training framework:

@article{soomro2012ucf101,
  title={UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild},
  author={Soomro, Khurram and Zamir, Amir Roshan and Shah, Mubarak},
  journal={arXiv preprint arXiv:1212.0402},
  year={2012}
}

@inproceedings{tran2018closer,
  title={A Closer Look at Spatiotemporal Convolutions for Action Recognition},
  author={Tran, Du and Wang, Heng and Torresani, Lorenzo and Ray, Jamie and LeCun, Yann and Paluri, Manohar},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2018}
}

@misc{saksena2025r3dhac,
  author = {Saumya Saksena},
  title = {{R3D-18 for UCF-101 Action Recognition}},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dronefreak/r3d-18-ucf101}},
  note = {Trained with the human-action-classification framework, Top-1 Accuracy: 83.43\%}
}

@software{saksena2026hac,
  author       = {Saumya Saksena},
  title        = {{Human Action Classification: Pose-based and Video-based Models}},
  year         = 2026,
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/dronefreak/human-action-classification}}
}

---

License

Apache-2.0