Whisper-Vad-EncDec-ASMR — torch weights

The same model as TransWithAI/Whisper-Vad-EncDec-ASMR-onnx, as a torch state_dict in safetensors, plus the module that runs it.

This is a format conversion, not a retrain. Same 29.77M fp32 parameters, same mel front-end, same architecture. On a 102-minute, 7-track reference set it reproduces the ONNX backend's output exactly: 830 of 830 intervals, 1660 of 1660 boundaries bit-identical, and 559 of 559 downstream ASR windows unchanged. The evidence is in parity.md.

Why this exists

The upstream model is published as ONNX only — there is no torch checkpoint on the Hub and none in the training repository. If you want this VAD on a GPU and you are already using torch, your options were to install onnxruntime-gpu next to it or to run the VAD on CPU.

The first is a cuDNN version fight: onnxruntime-gpu 1.19+ wants cuDNN 9, torch 2.2.x ships 8.8, and onnxruntime-gpu 1.18.1 fails DLL initialisation against that pairing on Windows. Moving the whole stack together to satisfy both is a real afternoon.

The second is slow. On one RTX 3070 Laptop, over 6130 s of audio:

So: 10.7× on the VAD stage, by removing a dependency rather than upgrading one. If torch's CUDA already works, this works. The ONNX export also pinned its output to batch 1, so it had to get its parallelism from threads; this takes any batch size.

Use fp32 unless you have checked that fp16 is safe for your pipeline — see Precision below. The fp16 row is in the table because it is a real option, not because it is the default.

Usage

pip install torch transformers safetensors numpy
hf download Raymxnd/Whisper-Vad-EncDec-ASMR-torch --local-dir ./whisper-vad-torch

vad_torch.py ships in this repo and is imported directly — the model is a plain nn.Module, not an AutoModel, so there is no trust_remote_code path.

import sys, librosa, torch
sys.path.insert(0, "./whisper-vad-torch")
from vad_torch import TorchVadRunner

runner = TorchVadRunner(
    weights="./whisper-vad-torch/model.safetensors",
    device="cuda",
    dtype=torch.float32,
)

audio, _ = librosa.load("speech.wav", sr=16000, mono=True)
for start, end in runner.speech_intervals(audio):
    print(f"{start:8.2f} -> {end:8.2f}")

speech_intervals handles 30 s chunking, thresholding, and merging across chunk seams. For raw frame logits, call runner.logits(chunks) with a list of 30 s arrays; it returns [N, 1500], one logit per 20 ms.

See example.py for a runnable version that also writes a .vtt of the detected speech regions.

Preprocessing is yours

The model takes 16 kHz mono float audio and this repo does not preprocess for you. The parity figures were measured with both backends fed the identical preprocessed waveform, which is what makes them a comparison of models rather than of pipelines. What you do upstream of that — resampling, loudness normalisation, channel mixing — is your pipeline's decision and will change the intervals for either backend equally.

How much it matters, concretely: on the first reference track, example.py's plain librosa.load(mono=True) finds 71 speech regions where the preprocessing used for parity.md finds 70. Same model, same weights, same threshold — a different mono mix. So do not read a region count off this model and treat it as a property of the model.

If your source is binaural or otherwise stereo, that is the mechanism to watch: a naive L/R average can cancel content that sits out of phase. Choosing between a mid mix and a side-emphasising mix based on channel correlation is worth measuring for your material — the reference set's per-track correlation runs from 0.05 to 0.78, and the right mix is not the same at both ends.

Precision

fp32 with TF32 disabled is the default, and that is measured, not assumed.

Every setting reproduces all 830 intervals, so an interval-count check passes either way and tells you nothing. fp16's boundaries are 99.3% exact, which also reads as fine. The column that matters is the last one: a boundary that shifts 0.220 s can flip a downstream 10 s window-splitting decision, which hands ~1.6 s of audio to a different ASR window. If nothing downstream of the VAD is sensitive to that, fp16 is 2.1× faster and a reasonable trade.

TF32 is the other half of the default. Ampere and later silently run fp32 matmul and convolution at a 10-bit mantissa; load_vad turns that off when you ask for fp32, and that is what closes the last gap — for free, since it measured the same speed either way.

Architecture

Read off the ONNX graph, not guessed. The details that are easy to get wrong:

• The encoder is fine-tuned whisper-base geometry, not whisper-base weights. Substituting openai/whisper-base runs fine and produces plausible, wrong intervals. Only the mel front-end is taken from openai/whisper-base; none of its weights are.
• The decoder is 2 × nn.TransformerDecoderLayer, post-norm, ReLU feed- forward — not GELU.
• frame_pos_embed is added to the encoder output to form the decoder's input; the decoder then cross-attends against the raw encoder output. It is not a standalone query set. Treating it as one runs, converges on plausible logits, and is wrong by a mean of 3.3.
• The head emits logits. There is no sigmoid in the graph, so the default threshold=0.4 is a threshold in logit space (≈0.60 after a sigmoid).

136 tensors, 29.77 M parameters, all fp32. load_vad is strict in both directions: every parameter must be present and every tensor consumed, because a silent partial load is exactly what produces plausible wrong intervals.

Provenance

extract_weights.py is the script that recovered these weights from the ONNX graph, included so the conversion is reproducible rather than asserted. It needs onnx and reads model.onnx from the upstream repo.

Two things made the extraction awkward and are handled there: the dynamo exporter renamed every weight matrix to val_N while leaving biases and LayerNorm scales under their torch names, so matrices are mapped back by walking the graph from the bias each one feeds; and the encoder's six k_proj matrices are bias-free in Whisper, so they are recovered by position.

Licence and attribution

MIT, inherited from upstream. The weights are derived from TransWithAI/Whisper-Vad-EncDec-ASMR-onnx by TransWithAI, whose training code, configs and ONNX export utilities are at TransWithAI/whisper-vad. If this model is useful to you, the credit for it belongs there; this repository contributes a runtime, not a model.

The upstream model builds on Whisper (arXiv:2212.04356) and on WhisperSeg's finding that Whisper's speech representations transfer to segmentation tasks.