Sagarmatha V6 — State-of-the-Art Nepali Speech Recognition

सगरमाथा V6: नेपाली भाषाको लागि अत्याधुनिक र नियन्त्रणयोग्य वाणी पहिचान प्रणाली

FLEURS ne_np 23.84 Raw WER · Common Voice 17 Test 23.40 Raw WER · 52× Real-Time Inference

Fine-tuned whisper-large-v3-turbo (809M) on 1,285.4 hours of multi-channel aligned Nepali speech, with 2,000 extended Devanagari BPE tokens and controllable script & style decoding.

---

Overview

Sagarmatha V6 is a state-of-the-art Automatic Speech Recognition (ASR) foundation model specifically engineered for the Nepali language and its complex dialectal, conversational, and acoustic realities.

Built on OpenAI's whisper-large-v3-turbo architecture, Sagarmatha V6 was trained on 1,285.4 hours of multi-channel Nepali audio (280,376 time-aligned windows) spanning broadcast news, conversational podcasts, television interviews, educational lectures, and field recordings across Nepal. The model features:

1. Extended Devanagari Vocabulary (+2,000 tokens): Drastically reduces sequence lengths and byte-fallback rates on Devanagari script, mitigating repetition loops and boosting inference throughput.
2. Controllable Multi-Mode Decoding: Dedicated decoder prefix tokens allowing users to dynamically select Script (all Devanagari vs mixed Latin/Devanagari) and Style (clean editorial vs verbatim with vocables).
3. Weight Averaged Soup ($\eta = 0.80$): Linearly interpolated with Sagarmatha V5 to prevent catastrophic forgetting and preserve robust out-of-distribution acoustic generalisation.
4. Dual Native Formats: Includes standard PyTorch weights (model.safetensors, 3.25 GB) and pre-converted float16 CTranslate2 weights (ct2-fp16/model.bin, 1.62 GB) delivering over 52× real-time throughput on modern GPUs.

---

Benchmark Results

All benchmarks are evaluated on Raw WER and Raw CER (standard Unicode NFC normalization with punctuation stripped, no artificial orthographic folding). 95% bootstrap confidence intervals are computed over clips (2,000 resamples).

Zero Leakage Guarantee: Rigorous n-gram cross-matching against the training corpus verifies that 0 benchmark sentences in FLEURS or Common Voice appear in the 1,285.4-hour training dataset.

1. Public Benchmarks (deva+clean mode)

FLEURS with digits spelled as words: 20.71 WER on the 588 digit-free reference subset.

---

2. Comparison with Contemporary Systems

---

Controllable Mode Tokens

Sagarmatha V6 introduces four steering prefix tokens placed immediately following <|startoftranscript|>:

[<|startoftranscript|>, <SCRIPT_TOKEN>, <STYLE_TOKEN>, <|ne|>, <|transcribe|>, <|notimestamps|>]

Recommended Production Prefix

For general-purpose production, subtitles, voice typing, and business transcription, use deva+clean:

• Token IDs: [53866, 53869, 50313, 50360, 52364]

---

Usage & Code Examples

1. Fast Inference with CTranslate2 (Recommended for Production)

CTranslate2 delivers 52× real-time processing with low memory consumption:

import ctranslate2
from transformers import WhisperProcessor
import librosa

# 1. Load processor and CTranslate2 model
processor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")
model = ctranslate2.models.Whisper(
    "tonibirat/sagarmatha-v6-nepali-asr",
    subfolder="ct2-fp16",
    device="cuda",  # or "cpu"
    compute_type="float16"  # or "int8_float16" on CUDA, "int8" on CPU
)

# 2. Load audio (resampled to 16 kHz mono)
audio, sr = librosa.load("nepali_speech.wav", sr=16000)

# 3. Extract log-mel filterbank features
inputs = processor(audio, sampling_rate=16000, return_tensors="np")
features = ctranslate2.StorageView.from_array(inputs.input_features)

# 4. Define mode prompt: <|deva|><|clean|><|ne|><|transcribe|><|notimestamps|>
# IDs: 53866 (<|deva|>), 53869 (<|clean|>), 50313 (<|ne|>), 50360 (<|transcribe|>), 52364 (<|notimestamps|>)
prompt_tokens = [53866, 53869, 50313, 50360, 52364]

# IMPORTANT: Always set max_length=896 to prevent Devanagari byte fallback cutoff
results = model.generate(
    features,
    [prompt_tokens],
    max_length=896,
    repetition_penalty=1.1,
    no_repeat_ngram_size=6
)

# 5. Decode predicted token IDs to Nepali text
output_ids = results[0].sequences_ids[0]
transcript = processor.tokenizer.decode(output_ids, skip_special_tokens=True)
print("Transcript:", transcript)

---

2. Standard Inference with Hugging Face Transformers

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import librosa

device = "cuda" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

# 1. Load model and processor
processor = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v6-nepali-asr")
model = WhisperForConditionalGeneration.from_pretrained(
    "tonibirat/sagarmatha-v6-nepali-asr",
    torch_dtype=torch_dtype
).to(device)

# 2. Load and process audio
audio, sr = librosa.load("nepali_speech.wav", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device, dtype=torch_dtype)

# 3. Force mode tokens in decoder: deva+clean
# <|deva|> (53866), <|clean|> (53869), <|ne|> (50313), <|transcribe|> (50360), <|notimestamps|> (52364)
forced_prefix_ids = [
    (1, 53866),  # <|deva|>
    (2, 53869),  # <|clean|>
    (3, 50313),  # <|ne|>
    (4, 50360),  # <|transcribe|>
    (5, 52364),  # <|notimestamps|>
]

# 4. Generate
with torch.no_grad():
    predicted_ids = model.generate(
        inputs.input_features,
        forced_decoder_ids=forced_prefix_ids,
        max_length=448
    )

transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print("Transcript:", transcript)

---

Critical Engineering Best Practices

1. Token Budget & Context Limit (max_length = 896)

In Whisper, each 30-second window is allotted a maximum context length of 448 positions. Under stock byte fallback or high-density Devanagari text, a 25-30 second Nepali audio window can generate up to 350+ tokens. When using CTranslate2, OpenAI's default implementation caps output at max_length // 2 (224 tokens), which cuts off 96.8% of long Nepali utterances. Always supply max_length = 896 in CTranslate2.

2. Voice Activity Detection (VAD) Gating

Like all sequence-to-sequence audio models, unconstrained decoding on long silent intervals or background music tracks can induce hallucinated text. For long-form audio or live streaming, always pair Sagarmatha V6 with a pre-segmentation VAD filter such as Silero VAD v5 (threshold=0.5). Gating removes over 95% of non-speech hallucination windows on challenging environmental audio (ESC-50).

3. Caution Regarding Whisper's BasicTextNormalizer

Do NOT use Whisper's stock BasicTextNormalizer when evaluating Devanagari ASR models. Its regex patterns treat Devanagari combining vowel signs (matras) and viramas as standalone whitespace/punctuation, exploding single Nepali words into 2.49× more sub-units and distorting WER results (spuriously inflating scores down to ~11.5%). Benchmark using standard Unicode NFC normalization with punctuation removal.

---

Training Details

• Base Architecture: openai/whisper-large-v3-turbo (809M parameters, 4 decoder layers, 32 encoder layers).
• Corpus: tonibirat/nepali-asr-v6 — 1,285.4 hours (280,376 audio windows) with word-level forced alignment.
• Hardware: NVIDIA RTX 5090 (32 GB VRAM).
• Optimizer & Schedule: AdamW, cosine learning rate decay with linear warmup, fp16 mixed precision.
• Soup Interpolation: $\theta_{\text{final}} = 0.80 \times \theta_{\text{V6}} + 0.20 \times \theta_{\text{V5}}$.

---

Citation & Contact

If you use Sagarmatha V6 in your research or application, please cite:

@misc{sagarmatha2026v6,
  author = {Basyal, Toni and contributors},
  title = {Sagarmatha V6: Controllable State-of-the-Art Speech Recognition for Nepali},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/tonibirat/sagarmatha-v6-nepali-asr}}
}

License

This model is licensed under the Apache License 2.0.