Oracolo Model Card

Model Description

Oracolo is a DeBERTa-based content moderation model trained to detect harmful content across multiple safety categories in Italian and English. The model analyzes both questions and answers in conversational contexts to determine potential safety risks.

Intended Use

• Content moderation for conversational AI systems
• Safety classification of question-answer pairs
• Multi-label classification across safety categories

Training Data

The model was trained on the BeaverTails dataset, which contains labeled examples of safe and unsafe conversational content.

Model Architecture

• Base model: Microsoft DeBERTa
• Multi-label classification head
• Maximum sequence length: 512 tokens

Performance

Overall Safety Classification

Comparison with PKU-Alignment/beaver-dam-7b on test set:

Usage Example

  from transformers import AutoTokenizer, AutoModelForSequenceClassification
  import torch
  def preprocess_text(prompt, response=""):
    """Format text in the same way as during training."""
    return f"<prompt> {prompt} </prompt> <response> {response} </response>"

  model = AutoModelForSequenceClassification.from_pretrained("path/to/oracolo")
  tokenizer = AutoTokenizer.from_pretrained("path/to/oracolo")
  model.eval()
  prompt = "How do I make a walkway slippery?"
  response = "I cannot provide advice that could lead to harm."
  formatted_text = preprocess_text(prompt, response)
  inputs = tokenizer(formatted_text, return_tensors="pt", truncation=True, max_length=512)
  with torch.no_grad():
    outputs = model(inputs)
    predictions = torch.sigmoid(outputs.logits).cpu().numpy()[0]
    # Apply threshold (0.3 recommended based on validation)
    class_predictions = (predictions > 0.3).astype(int)

Full classification report