Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages

📚 Transformers docs | 📃 Paper | 📝 Blogpost | 📄 License: Apache 2.0

Model Description

This is a 🤗 Transformers-compatible conversion of omniASR_LLM_300M_v2, part of the Omnilingual ASR family released by Meta AI, which covers more than 1,600 languages — including hundreds never previously supported by any ASR system.

This checkpoint is an LLM (audio language model) variant: the audio embeddings of a Wav2Vec2-style encoder are linearly projected into a Llama decoder which autoregressively generates the transcription. This variant supports optional language conditioning: passing a language code such as "eng_Latn" (or "en") writes the matching language token into the decoder context, which generally improves transcription quality.

The Transformers-compatible checkpoints are:

• Developed by: Meta AI / Omnilingual ASR Team (GitHub)
• Converted to Transformers by: Eric Bezzam
• Model type: Wav2Vec2-style speech encoder + Llama decoder (audio language model), autoregressive decoding.
• Language(s): 1,600+ languages (see Supported languages).
• License: Apache-2.0 (for the model and code), CC-BY-4.0 for the facebook/omnilingual-asr-corpus dataset.

---

Installation

OmniASR is not yet part of a Transformers release, so install Transformers from source:

pip install git+https://github.com/huggingface/transformers

Usage

[!NOTE] The simplest way to transcribe audio is with apply_transcription_request, which handles the chat template formatting for you, namely it is a convenience wrapper for apply_chat_template (see Chat template below).

Simple transcription

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "bezzam/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
)
inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(processor.decode(generated_ids, skip_special_tokens=True)[0])

Language hint

Without a language, the model runs in its language-agnostic mode. Passing a language code (e.g. "eng_Latn", "fra_Latn", "cmn_Hans") writes the matching language token into the decoder prompt, which generally improves transcription quality. The list of supported languages and naming convention follows the original here. For the languages that have one, the more common ISO 639-1 codes (e.g. "en", "fr", "zh") are also accepted.

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "bezzam/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

audio = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav"

# Without language hint (language-agnostic)
inputs = processor.apply_transcription_request(audio=audio).to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(f"Language-agnostic: {processor.decode(generated_ids, skip_special_tokens=True)[0]}")

# With language hint
inputs = processor.apply_transcription_request(audio=audio, language="zh").to(model.device, dtype=model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(f"Language hint:     {processor.decode(generated_ids, skip_special_tokens=True)[0]}")

Batch inference

Batch inference is possible by passing a list of audios and, if provided, a single language for the whole batch or a list of languages (one per audio, None for the language-agnostic mode).

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "bezzam/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

audio = [
    "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
]

inputs = processor.apply_transcription_request(audio, language=["en", None])
inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
for i, text in enumerate(processor.decode(generated_ids, skip_special_tokens=True)):
    print(f"Audio {i + 1}: {text}")

Chat template

The model also accepts chat template inputs. The apply_transcription_request usage above is a convenience wrapper for apply_chat_template. The language hint is given as a {"type": "language", ...} item of the user turn; leaving it out selects the language-agnostic mode.

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "bezzam/omniasr-llm-300m-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

conversations = [
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
                },
                {"type": "language", "language": "en"},
            ],
        },
    ],
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mandarin_voxcpm_zh.wav",
                },
                {"type": "language", "language": "zh"},
            ],
        },
    ],
]

inputs = processor.apply_chat_template(
    conversations, tokenize=True, add_generation_prompt=True, return_dict=True
)
inputs.to(model.device, dtype=model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
for i, text in enumerate(processor.decode(generated_ids, skip_special_tokens=True)):
    print(f"Audio {i + 1}: {text}")

Training

The model can be trained with the loss it outputs. Pass the target transcripts to apply_transcription_request with transcription, which writes them as the assistant turn and returns labels. Everything but the transcript and its EOS (the audio placeholders, the language prompt and the padding) is masked automatically. With apply_chat_template, the same is done by putting the transcript in an assistant turn and passing processor_kwargs={"output_labels": True}. Note that OmniASR transcribes in lowercase, so the (uppercase) LibriSpeech transcripts are lowercased below.

from datasets import Audio, load_dataset
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "bezzam/omniasr-llm-300m-v2"
NUM_SAMPLES = 5

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
model.train()

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate))
speech_samples = [el["array"] for el in ds["audio"][:NUM_SAMPLES]]
text_samples = [text.lower() for text in ds["text"][:NUM_SAMPLES]]

# Any supported language code, or `language=None` to train the language-agnostic mode
inputs = processor.apply_transcription_request(speech_samples, language="en", transcription=text_samples)
inputs.to(model.device, dtype=model.dtype)

outputs = model(**inputs)
print("Loss:", outputs.loss.item())
outputs.loss.backward()

Torch compile

For autoregressive transcription, torch.compile accelerates the per-token forward passes inside generate by providing a CompileConfig object, which requires a static cache.

Note that the StaticCache is created once and reused across generate calls to avoid recompiles.

On an A100, we observed a speed-up of ~2.3 for a batch size of 4 (script).

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor, CompileConfig, StaticCache

model_id = "bezzam/omniasr-llm-300m-v2"
NUM_SAMPLES = 4
num_warmup = 3
max_new_tokens = 256

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = processor.apply_transcription_request(
    audio=[audio_url] * NUM_SAMPLES,
    language="en",
).to(model.device, dtype=model.dtype)

# Created once and reused across `generate` calls, to avoid recompiling at every call
static_cache = StaticCache(
    config=model.config,
    max_cache_len=inputs["input_ids"].shape[1] + max_new_tokens,
)
compile_config = CompileConfig()

# Warmup
with torch.inference_mode():
    for _ in range(num_warmup):
        static_cache.reset()
        _ = model.generate(
            **inputs, max_new_tokens=max_new_tokens, do_sample=False,
            past_key_values=static_cache, compile_config=compile_config,
        )
torch.cuda.synchronize()

# Apply model
with torch.inference_mode():
    static_cache.reset()
    output_ids = model.generate(
        **inputs, max_new_tokens=max_new_tokens, do_sample=False,
        past_key_values=static_cache, compile_config=compile_config,
    )
generated_ids = output_ids[:, inputs["input_ids"].shape[1] :]
print(processor.decode(generated_ids, skip_special_tokens=True)[0])

Supported Languages

The list of 1600+ supported languages and their naming convention follows the original here. Languages follow the format {language_code}_{script}, for example eng_Latn - English (Latin script), cmn_Hans - Mandarin Chinese (Simplified), ...

---

Citation

BibTeX:

@misc{omnilingualasr2025,
  title={{Omnilingual ASR}: Open-Source Multilingual Speech Recognition for 1600+ Languages},
  author={{Omnilingual ASR Team} and Keren, Gil and Kozhevnikov, Artyom and Meng, Yen and Ropers, Christophe and Setzler, Matthew and Wang, Skyler and Adebara, Ife and Auli, Michael and Can, Balioglu and Chan, Kevin and Cheng, Chierh and Chuang, Joe and Droof, Caley and Duppenthaler, Mark and Duquenne, Paul-Ambroise and Erben, Alexander and Gao, Cynthia and Mejia Gonzalez, Gabriel and Lyu, Kehan and Miglani, Sagar and Pratap, Vineel and Sadagopan, Kaushik Ram and Saleem, Safiyyah and Turkatenko, Arina and Ventayol-Boada, Albert and Yong, Zheng-Xin and Chung, Yu-An and Maillard, Jean and Moritz, Rashel and Mourachko, Alexandre and Williamson, Mary and Yates, Shireen},
  year={2025},
  url={https://ai.meta.com/research/publications/omnilingual-asr-open-source-multilingual-speech-recognition-for-1600-languages/},
}