FaceScanPaliGemma_Emotion

from PIL import Image
import torch
from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig, TrainingArguments, Trainer


model = PaliGemmaForConditionalGeneration.from_pretrained('NYUAD-ComNets/FaceScanPaliGemma_Emotion',torch_dtype=torch.bfloat16)

input_text = "what is the emotion of the person in the image?"

processor = PaliGemmaProcessor.from_pretrained("google/paligemma-3b-pt-224")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model.to(device)


input_image = Image.open('image_path')
inputs = processor(text=input_text, images=input_image, padding="longest", do_convert_rgb=True, return_tensors="pt").to(device)
inputs = inputs.to(dtype=model.dtype)
      
with torch.no_grad():
          output = model.generate(**inputs, max_length=500)
result=processor.decode(output[0], skip_special_tokens=True)[len(input_text):].strip()

Model description

This model is a fine-tuned version of google/paligemma-3b-pt-224 on the AffectNet dataset. The model aims to classify the emotion of face image or image with one person into eight categoris such as 'neutral', 'happy', 'sad', 'surprise', 'fear', 'disgust', 'anger', 'contempt'

Model Performance

Accuracy: 59.4 %, F1 score: 59 %

Intended uses & limitations

This model is used for research purposes

Training and evaluation data

AffectNet dataset was used for training and validating the model

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 2
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 2
• num_epochs: 5

Training results

Framework versions

• Transformers 4.42.4
• Pytorch 2.1.2+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1

BibTeX entry and citation info

@article{aldahoul2026facescanpaligemma,
  title={FaceScanPaliGemma multi-agent vision language models for facial attribute recognition},
  author={AlDahoul, Nouar and Tan, Myles Joshua Toledo and Kasireddy, Harishwar Reddy and Zaki, Yasir},
  journal={Scientific Reports},
  year={2026},
  publisher={Nature Publishing Group UK London}
}

@article{aldahoul2024exploring,
  title={Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age},
  author={AlDahoul, Nouar and Tan, Myles Joshua Toledo and Kasireddy, Harishwar Reddy and Zaki, Yasir},
  journal={arXiv preprint arXiv:2410.24148},
  year={2024}
}

@misc{ComNets,
      url={https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion](https://huggingface.co/NYUAD-ComNets/FaceScanPaliGemma_Emotion)},
      title={FaceScanPaliGemma_Emotion},
      author={Nouar AlDahoul, Yasir Zaki}
}

Governance & Responsible Use

The FaceScanPaliGemma model processes highly sensitive biometric data (facial attributes). Deployment of this model must follow strict governance frameworks to ensure responsible and ethical use.

✅ Permitted Uses

• Academic research, benchmarking, and reproducibility studies.
• Educational projects exploring bias, fairness, and multimodal AI.
• Development of fairness-aware systems with proper safeguards.

❌ Prohibited Uses

• Surveillance or mass monitoring of individuals or groups.
• Identity verification or authentication without explicit and informed consent.
• Applications that discriminate against or marginalize individuals or communities.
• Use on scraped datasets or facial images collected without consent.

⚠️ Law Enforcement Use

• Direct use in law enforcement contexts is not recommended due to high societal risks.
• Risks include bias amplification, wrongful identification, and privacy violations.
• If ever considered, deployment must be:
  • Governed by strict legal frameworks (e.g., EU AI Act, GDPR, CCPA).
  • Subject to independent auditing, transparency, and accountability.
  • Limited to proportional, necessary, and rights-respecting use cases.

Governance Principles

1. Access & Control – Limit deployment to contexts with clear oversight and accountability.
2. Transparency – Always disclose when and how the model is used.
3. Bias & Fairness Auditing – Evaluate performance across demographic groups before deployment.
4. Privacy Protection – Respect GDPR, CCPA, and local regulations; never process data without consent.
5. Accountability – Establish internal review boards or ethics committees for production use.

Community Reporting

We encourage the community to report issues, biases, or misuse of this model through the Hugging Face Hub discussion forum.