Svarupa ASR 0.6B v2 EOT (svarupa_asr_0.6b_v2_EOT)

Streaming end-of-turn detection on top of Svarupa ASR 0.6B v2. The ASR encoder stays frozen. A 664k-parameter head reads three encoder layers and, every 80 ms, classifies the frame as speech, hold (the speaker paused but will continue), or end of turn.

Transcription and end-of-turn detection are two reads of the same encoder pass. The decoder produces the transcript. The head produces the turn decision. They do not feed each other.

What is in this repo

The 0.6B FastConformer itself is not duplicated here. It is downloaded from sampathlonka/svarupa_v2-asr-0.6b.

Install

pip install "nemo_toolkit[asr]==3.0.0" torch huggingface_hub numpy

NeMo 3.0 matches the base checkpoint. Audio is 16 kHz mono.

Load

from modeling import SvarupaEoT

model = SvarupaEoT.from_pretrained("sampathlonka/svarupa_asr_0.6b_v2_EOT")

from_pretrained downloads this repo and svarupa_v2-asr-0.6b.nemo. Pass token= if you need a private base, and device="cpu" or "cuda".

Transcribe

hyps = model.transcribe("utterance.wav")
text = hyps[0].text if hasattr(hyps[0], "text") else hyps[0]

This is the frozen Svarupa v2 recognizer. Decoding defaults are the ones stored in the .nemo. For beam-4 decoding, follow the quickstart on the base model card.

End of turn

import soundfile as sf

audio, sr = sf.read("utterance.wav", dtype="float32")
assert sr == 16000

probs, events = model.endpoint(audio)
# probs[t] = [p_speech, p_hold, p_eot] for the 80 ms frame ending at (t + 1) * 0.08 s
for event in events:
    print(event.kind, round(event.time_s, 2), round(event.p_eot, 3))

endpoint walks the audio in 80 ms cache-aware chunks. A pause starts when p_speech drops below 0.5. After the pause has lasted 240 ms, the turn ends at the first frame with p_eot >= 0.628. If that never happens, a 1.0 s fallback closes the turn. Those three numbers are the operating point chosen on held-out real calls (see Evaluation).

If you already have pause boundaries from your own VAD, score them with decide. That is the rule behind the table below, and it is the setup to use in production. The head is better at judging a pause than at finding one.

probs, _ = model.endpoint(audio)
event = model.decide(probs[:, 2], pause_start_s=1.40, pause_end_s=3.00)
# event.kind is "eot" or "fallback", or None if the pause ended first

Using both at once

Keep one loaded SvarupaEoT. Each live chunk goes through the encoder once.

• The encoder output continues into the RNNT decoder and updates the partial transcript (transcribe does this for a finished utterance).
• Layers 8, 16 and 24 of that same forward pass go into the head (endpoint).

When an eot or fallback event fires, finalize the partial transcript and hand the turn to the agent. Reset the streaming caches after the turn. Do not reset them in the middle of one.

The language prompt is attached after the encoder, so the head does not depend on Hindi, English, or Hinglish prompting. Set the prompt for the transcript the same way you do for Svarupa v2.

The encoder lookahead must stay [56, 0]. The head was trained on causal 80 ms frames. A larger right context changes the features and the head is no longer valid. Retrain the head if you need a bigger ASR lookahead.

Architecture

16 kHz audio
    └─ frozen FastConformer (24 layers, 1024-d, 80 ms frames, lookahead 0)
           ├─ top encoder state ── RNNT decoder ── transcript
           └─ layers 8, 16, 24
                 LayerNorm, learned mix 0.37 / 0.38 / 0.26
                 Linear 1024 → 256, GELU
                 GRU 256
                 Linear → speech / hold / end-of-turn

The head has 664,070 parameters. Training updated only those weights.

Frame targets: leading silence is ignored, speech stays speech, a pause where the same speaker resumes is hold, and the silence after a finished turn is end of turn.

Operating point

Chosen on one half of the real-call eval set. The premature budget was the share of turns a 1 s timeout cuts by itself, plus 5 points.

Evaluation

Held-out speakers. Latency is the time after the speaker stops, measured only on turns the head caught before the 1 s fallback. "Detected" is the share of true ends of turn caught before that fallback.

Real Hindi and English calls (Basis, 4,869 clips)

About 11.4 points of the 16.4% premature rate are pauses longer than 1 s, which the fallback cuts no matter what the head says. The head itself adds about 5 points.

Other sets

Training

The encoder was not updated. The head was trained for 6 epochs, AdamW at 1e-3, batch size 256, on a 70 / 30 mix of:

• Basis Conversations 1500, Hindi and English channel-separated calls. A turn is an end of turn when another speaker replies within 2 s, and a hold when the same speaker resumes within 0.2–1.3 s. 276,781 training clips after trimming each clip to the current turn.
• Smart Turn v3.2, the Hindi and English subset. These clips teach whether a sentence sounds finished. Every clip ends with the same amount of silence, so timing alone cannot solve them.

Best validation checkpoint. Learned layer mix: layer 8 = 0.370, layer 16 = 0.375, layer 24 = 0.256.

Limitations

• Causal 80 ms chunks only. Do not pair this head with a larger encoder lookahead.
• Judging a known pause is the supported task. Finding the pause from p_speech alone is weaker. Pass pauses from Silero or from the recognizer's blanks into decide.
• About a third of the real-call end-of-turn labels are soft: overlapping replies, replies after 1.3 s, or audio the transcript never turned into words. That caps accuracy on calls.
• Hindi is detected more often than English and is cut off more often. One threshold serves both languages.
• No pitch features and no transcript features are used. Local word identity ("yeah", "okay", "हाँ") does not say whether the speaker will continue.

License

Apache 2.0. The head is trained on top of sampathlonka/svarupa_v2-asr-0.6b, which is fine-tuned from nvidia/nemotron-3.5-asr-streaming-0.6b.