This is an ONNX conversion of Oriserve/Whisper-Hindi2Hinglish-Swift at revision cd0d4d013062d542569135460a79025337df1982, for Transformers.js with word-level timestamps. It runs on the CPU (WASM) in the browser and powers the on-device "Hinglish (Roman letters)" captions in Cutmallow.
https://cutmallow.com/
The model is a whisper-base fine-tune (73 M parameters) by Oriserve, trained on about 550 hours of Indian-accented Hindi. It writes Hindi/English code-mixed speech as romanized Hinglish, and English words stay in Latin script. Prompt it with the English language token.
| File | Notes |
|---|---|
onnx/encoder_model.onnx |
fp32 (82.4 MB). An 8-bit encoder lost accuracy, so it stays fp32. |
onnx/decoder_model_merged_quantized.onnx |
8-bit merged decoder (52.8 MB). It outputs cross_attentions.* for word timestamps. The logits projection reuses the 8-bit token embedding (tied weights). There is no cache_position input. |
generation_config.json |
Adds whisper-base alignment_heads (the fine-tune ships none). Word timing agrees with whisper-base within 20–40 ms (median). |
import { pipeline } from '@huggingface/transformers';
const asr = await pipeline('automatic-speech-recognition', 'di12355/whisper-hinglish-swift-onnx', {
device: 'wasm',
dtype: { encoder_model: 'fp32', decoder_model_merged: 'q8' },
});
const out = await asr(float32Audio16kHz, { language: 'en', task: 'transcribe', return_timestamps: 'word' });
Pass audio in speech windows of 30 s or less.
| Model | WER |
|---|---|
whisper-base, hi |
84.2% |
whisper-base, en (translates) |
78.6% |
| This model | 22.0% |
The conversion uses the pinned toolchain torch 2.5.1, transformers 4.48.3, optimum 1.24.0, onnx 1.17.0 and onnxruntime 1.20.1:
main_export (automatic-speech-recognition-with-past, opset 14) with a Whisper ONNX config that adds cross_attentions.{0..5} decoder outputs and drops cache_position.onnxruntime.quantization.quantize_dynamic on the merged decoder (QInt8, MatMul + Gather, EnableSubgraph).MatMulInteger.The script is scripts/hinglish/convert.py in the Cutmallow repository.
Apache-2.0, the same as the upstream fine-tune (see LICENSE, which also carries the OpenAI Whisper MIT notice). All credit for the weights goes to Oriserve. This repo only converts them.
onnx/encoder_model.onnx 82 MBonnx/decoder_model_merged_quantized.onnx 53 MBOriserve/Whisper-Hindi2Hinglish-Swift