V-JEPA for Isolated Sign Language Recognition (vjepa-16f-25fps-384px-p2)

This model repository contains the fine-tuned weights for vjepa-16f-25fps-384px-p2, the best-performing model from the paper:

Sign Language Recognition with Video Foundation Models: V-JEPA Transfer and Adaptation
Giovanne Mariano and Helio Pedrini (Institute of Computing, University of Campinas - UNICAMP)
Paper / Repository: GitHub Code & Artifacts

---

Model Description

• Backbone: V-JEPA 2.1 (ViT-B)
• Head: Attentive Pooling Layer + Linear Classifier
• Task: Isolated Sign Language Recognition (ISLR) on French Belgian Sign Language (LSFB-isol) across 700 classes
• Resolution: 384x384 px
• Temporal Configuration: 16 context frames sampled at 25 FPS (sub-sampled from native 50 FPS)
• Training Strategy: Two-phase training - initial linear head pre-training followed by full encoder fine-tuning (p2).

---

Benchmark Results (700-Class LSFB-isol)

Evaluated under a strict Signer-Independent (SI) split with held-out signers:

---

Experiment Configuration (config.yaml)

global:
  seed: 73
  model_name: vjepa
execution:
  batch_size: 32
  accumulate_grad_batches: 1
  num_workers: 4
  use_gradient_checkpointing: true
dataset:
  data_root: /path/to/lsfb-isol
  train_split: si_train
  val_split: si_val
  num_classes: 700
pipeline:
  max_frames: 80
  num_frames: 16
  crop_size: 384
  target_fps: 25.0
model:
  pretrained_path: vjepa2_1_vitb_dist_vitG_384.pt
  freeze_encoder: false
  vjepa:
    head_dropout: 0.4
optimizer:
  name: adamw
  lr: 3.3e-06
  weight_decay: 0.056
training:
  max_epochs: 50
  warmup_epochs: 5
  resume_from: best_weights_phase1.ckpt
  load_weights_from: null

---

Usage & Inference Example

To load the model weights in PyTorch:

import torch
from huggingface_hub import hf_hub_download

# Download weights from Hugging Face Hub
repo_id = "giolucasd/vjepa-16f-25fps-384px-p2"
weights_path = hf_hub_download(repo_id=repo_id, filename="best_weights.pt")

# Load state dict
state_dict = torch.load(weights_path, map_location="cpu", weights_only=True)

---

Citation

If you use this model or code in your research, please cite our work:

@inproceedings{mariano2026sign,
  title={Sign Language Recognition with Video Foundation Models: V-JEPA Transfer and Adaptation},
  author={Giovanne Mariano and H{\'e}lio Pedrini},
  booktitle={Workshop de Vis{\~a}o Computacional (WVC)},
  year={2026}
}