Multi-Label Audio Expressiveness Classifier

This model is a fine-tuned version of openai/whisper-small for multi-label audio classification. It detects stylistic traits in speech, allowing for multiple tags to be active simultaneously.

🏷️ Labels

• animated, laughing, passive, whispered, enunciated

---

🛠 Usage: Input & Output

1. Input Specifications

• Processor: Uses WhisperProcessor to handle audio resampling and Mel-spectrogram conversion.
• Sampling Rate: 16,000 Hz.
• Format: Raw waveform (Mono).

2. Output (Vector Interpretation)

This is a Multi-Label model. Instead of a single "winner," it outputs a probability for every tag independently.

• Activation: Sigmoid (probabilities 0.0 to 1.0).
• Decision: A tag is "Active" if its probability > Threshold (Default: 0.5).

---

🚀 Inference Code

import torch
import librosa
from transformers import WhisperProcessor, WhisperForAudioClassification

model_id = "Kang-Chieh/whisper-small-mlb-expressiveness"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Use WhisperProcessor (wraps feature extractor)
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForAudioClassification.from_pretrained(model_id).to(device)

def predict_expressiveness(audio_path, threshold=0.5):
    # 1. Load and resample
    audio, _ = librosa.load(audio_path, sr=16000)
    
    # 2. Preprocess with Processor
    inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device)

    # 3. Inference
    with torch.no_grad():
        logits = model(**inputs).logits
    
    # 4. Multi-label logic (Sigmoid)
    probs = torch.sigmoid(logits).squeeze().cpu().numpy()
    
    id2label = model.config.id2label
    results = {id2label[i]: float(probs[i]) for i in range(len(probs))}
    active_tags = [tag for tag, score in results.items() if score > threshold]
    
    return active_tags, results

tags, scores = predict_expressiveness("sample.wav")
print(f"Tags: {tags}")