Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages

📚 Transformers docs | 📃 Paper | 📝 Blogpost | 📄 License: Apache 2.0

Model Description

This is a 🤗 Transformers-compatible conversion of omniASR_CTC_3B_v2, part of the Omnilingual ASR family released by Meta AI, which covers more than 1,600 languages — including hundreds never previously supported by any ASR system.

This checkpoint is a CTC variant: a Wav2Vec2-style encoder (convolutional layers that downsample the 16 kHz waveform ~320× to a 50 Hz frame rate, followed by a pre-norm Transformer encoder) with a single linear projection to the vocabulary, decoded non-autoregressively with greedy CTC. It is the fastest variant, transcribes in lowercase without punctuation, and does not take a language hint.

The Transformers-compatible checkpoints are:

• Developed by: Meta AI / Omnilingual ASR Team (GitHub)
• Converted to Transformers by: Eric Bezzam
• Model type: Wav2Vec2-style speech encoder with a CTC head, non-autoregressive decoding.
• Language(s): 1,600+ languages (see Supported languages).
• License: Apache-2.0 (for the model and code), CC-BY-4.0 for the facebook/omnilingual-asr-corpus dataset.

---

Installation

OmniASR is not yet part of a Transformers release, so install Transformers from source:

pip install git+https://github.com/huggingface/transformers

Usage

The CTC variant transcribes in lowercase without punctuation, and does not take a language hint.

Simple transcription

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "bezzam/omniasr-ctc-3b-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))

inputs = processor(ds[0]["audio"]["array"], sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs)
print(processor.decode(outputs, skip_special_tokens=True)[0])

Batch inference

Batch inference is possible by passing a list of audios, which the processor pads to the longest one.

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "bezzam/omniasr-ctc-3b-v2"
NUM_SAMPLES = 5

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]

inputs = processor(speech_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs)
for i, text in enumerate(processor.decode(outputs, skip_special_tokens=True)):
    print(f"Audio {i + 1}: {text}")

Training

The model can be trained with the loss it outputs. Passing text to the processor prepares the CTC labels, with padding positions masked out. Note that OmniASR transcribes in lowercase, so the (uppercase) LibriSpeech transcripts are lowercased below.

from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor

model_id = "bezzam/omniasr-ctc-3b-v2"
NUM_SAMPLES = 5

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = [text.lower() for text in ds["text"][:NUM_SAMPLES]]

# passing `text` to the processor will prepare inputs' `labels` key
inputs = processor(audio=speech_samples, text=text_samples, sampling_rate=processor.feature_extractor.sampling_rate)
inputs.to(model.device, dtype=model.dtype)

outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()

Torch compile

The CTC variant runs a single forward pass, which is compiled by passing a CompileConfig to generate. Padding all inputs to the same length (here 30 seconds) keeps the input shapes static, so that new inputs don't trigger a recompilation.

On an A100, we observed a speed-up of ~1.3 for a batch size of 4 (script).

import torch
from datasets import Audio, load_dataset
from transformers import AutoModelForCTC, AutoProcessor, CompileConfig

model_id = "bezzam/omniasr-ctc-3b-v2"
NUM_SAMPLES = 4
num_warmup = 3

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForCTC.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]

inputs = processor(
    speech_samples,
    sampling_rate=processor.feature_extractor.sampling_rate,
    padding="max_length",
    max_length=30 * processor.feature_extractor.sampling_rate,
).to(model.device, dtype=model.dtype)

compile_config = CompileConfig(fullgraph=True, mode="reduce-overhead")

# Warmup
for _ in range(num_warmup):
    _ = model.generate(**inputs, compile_config=compile_config)
torch.cuda.synchronize()

# Apply model
outputs = model.generate(**inputs, compile_config=compile_config)
print(processor.decode(outputs, skip_special_tokens=True)[0])

Supported Languages

The list of 1600+ supported languages and their naming convention follows the original here. Languages follow the format {language_code}_{script}, for example eng_Latn - English (Latin script), cmn_Hans - Mandarin Chinese (Simplified), ...

---

Citation

BibTeX:

@misc{omnilingualasr2025,
  title={{Omnilingual ASR}: Open-Source Multilingual Speech Recognition for 1600+ Languages},
  author={{Omnilingual ASR Team} and Keren, Gil and Kozhevnikov, Artyom and Meng, Yen and Ropers, Christophe and Setzler, Matthew and Wang, Skyler and Adebara, Ife and Auli, Michael and Can, Balioglu and Chan, Kevin and Cheng, Chierh and Chuang, Joe and Droof, Caley and Duppenthaler, Mark and Duquenne, Paul-Ambroise and Erben, Alexander and Gao, Cynthia and Mejia Gonzalez, Gabriel and Lyu, Kehan and Miglani, Sagar and Pratap, Vineel and Sadagopan, Kaushik Ram and Saleem, Safiyyah and Turkatenko, Arina and Ventayol-Boada, Albert and Yong, Zheng-Xin and Chung, Yu-An and Maillard, Jean and Moritz, Rashel and Mourachko, Alexandre and Williamson, Mary and Yates, Shireen},
  year={2025},
  url={https://ai.meta.com/research/publications/omnilingual-asr-open-source-multilingual-speech-recognition-for-1600-languages/},
}