WhissleAI Gujlish wav2vec2 Meta-ASR

A bilingual English-Gujarati (Gujlish) speech recognition model that simultaneously transcribes speech and extracts rich metadata — speaker age, gender, emotion, intent, and named entities — in a single forward pass using CTC decoding.

Model Description

This model is fine-tuned from CLSRIL-23 (a wav2vec2-base model pre-trained on 23 Indian languages by IIT Madras) for bilingual English + Gujarati ASR with inline meta-tag prediction. It uses a unified character-level vocabulary with atomic meta-tag tokens, enabling the CTC head to output both transcript text and structured metadata.

Meta-Tags

The model predicts the following inline tags alongside the transcript:

Example output:

the weather in ahmedabad is very hot today ENTITY_LOCATION END AGE_30_45 GENDER_MALE EMOTION_NEUTRAL INTENT_INFORM

Performance

ASR (Word Error Rate)

• Full WER = WER on raw output (transcript + tags)
• Clean WER = WER on transcript only (tags stripped)

Meta-Tag Accuracy

Latency

Measured on NVIDIA A100 GPU, batch_size=1, average utterance ~5 seconds.

Usage

Quick Start (HuggingFace)

import torch
import torchaudio
from transformers import Wav2Vec2ForCTC
import json

# Load model
model = Wav2Vec2ForCTC.from_pretrained("WhissleAI/speech-tagger_gujlish_wav2vec2_meta")
model.eval()

# Load vocab
from huggingface_hub import hf_hub_download
vocab_path = hf_hub_download("WhissleAI/speech-tagger_gujlish_wav2vec2_meta", "vocab.json")
with open(vocab_path) as f:
    vocab = json.load(f)
id_to_token = {v: k for k, v in vocab.items()}

# Load audio
waveform, sr = torchaudio.load("test.wav")
if sr != 16000:
    waveform = torchaudio.functional.resample(waveform, sr, 16000)
waveform = waveform.mean(dim=0) if waveform.shape[0] > 1 else waveform.squeeze(0)

# Inference
with torch.no_grad():
    logits = model(input_values=waveform.unsqueeze(0)).logits

# CTC greedy decode
pred_ids = torch.argmax(logits, dim=-1)[0].tolist()
collapsed = []
prev = -1
for p in pred_ids:
    if p != prev and p != 0:  # 0 = <pad>/blank
        collapsed.append(p)
    prev = p

text = "".join(id_to_token.get(i, "") if id_to_token.get(i) != "|" else " " for i in collapsed).strip()
print(text)

Using the Inference Script

# Install dependencies
pip install torch torchaudio transformers huggingface_hub numpy

# Transcribe a single file
python infer_wav2vec2_gujlish.py --model WhissleAI/speech-tagger_gujlish_wav2vec2_meta --audio test.wav

# Batch transcribe from manifest
python infer_wav2vec2_gujlish.py \
    --model WhissleAI/speech-tagger_gujlish_wav2vec2_meta \
    --manifest test.json \
    --output results.json \
    --batch-size 16 \
    --device cuda

From PyTorch Lightning Checkpoint (.ckpt)

python infer_wav2vec2_gujlish.py \
    --model ./checkpoints/gujlish-wav2vec2-clsril23-v1-step=65000-val_wer=0.2610.ckpt \
    --pretrained ./CLSRIL-23.pt \
    --vocab ./vocab.json \
    --audio test.wav

Training Details

Data

Language split: 68.6% English, 31.4% Gujarati (INDO_ARYAN family).

All training samples were annotated with meta-tags (AGE, GENDER, EMOTION, INTENT, ENTITY) using Whissle's automated annotation pipeline powered by Gemini.

Training Configuration

Training Procedure

1. Pretrained checkpoint: CLSRIL-23 (fairseq format) → converted to HuggingFace Wav2Vec2ForCTC
2. Vocabulary: Built from training manifest — 795 tokens (characters for both scripts + meta-tags)
3. Feature extractor frozen: Only the transformer encoder and CTC head are fine-tuned
4. CTC loss: Standard CTC with blank token <pad> (id=0)
5. Per-language WER tracking: Separate validation metrics for English and Gujarati

Architecture

Raw Audio (16kHz)
       │
       ▼
┌──────────────────┐
│ Feature Extractor │  ← Frozen (7 conv layers, stride=320)
│ (CNN)            │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Feature Projection│  ← 512 → 768
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ Transformer      │  ← 12 layers, 768 hidden, 12 heads
│ Encoder          │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│ CTC Head         │  ← Linear(768 → 795)
│ (lm_head)        │
└────────┬─────────┘
         │
         ▼
  CTC Greedy Decode
         │
         ▼
"hello world AGE_30_45 GENDER_MALE EMOTION_NEUTRAL INTENT_INFORM"

Limitations

• Gujarati script: The model handles Gujarati script but WER is higher than English (23% vs 16% clean WER) due to less training data
• Entity extraction: Entity F1 is moderate for English (41%) and lower for Gujarati (14%) — primarily trained on person names and locations
• Code-switching: While the model handles English-Gujarati code-switching, rapid intra-sentential switching may increase WER
• Noise robustness: Trained with aggressive noise augmentation but performance degrades significantly below 5dB SNR
• Max duration: Trained on utterances up to 16 seconds; longer audio should be chunked

ONNX Export

An ONNX-exported version of this model is available for production deployment. The ONNX model supports dynamic batch size and sequence length:

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model.onnx")
audio = np.random.randn(1, 48000).astype(np.float32)  # 3 seconds
logits = session.run(None, {"input_values": audio})[0]

Citation

@misc{whissle2025gujlish,
  title={Gujlish Meta-ASR: Bilingual English-Gujarati Speech Recognition with Inline Metadata Prediction},
  author={Whissle AI},
  year={2025},
  url={https://huggingface.co/WhissleAI/speech-tagger_gujlish_wav2vec2_meta}
}

License

This model is released under the Apache 2.0 License.

About WhissleAI

WhissleAI builds production-grade speech AI systems. Our Meta-ASR models jointly perform speech recognition and speaker/content analysis in a single pass, enabling real-time applications like live coaching, meeting intelligence, and voice analytics.

• Website: whissle.ai
• GitHub: github.com/WhissleAI
• HuggingFace: huggingface.co/WhissleAI