Kana Whisper

Kana Whisper is a fine-tuned Whisper large-v3-turbo model that transcribes Japanese speech directly into katakana sequences.

This model is the ASR component of the Joyo Kanji Yomi Benchmark, a systematic evaluation framework for kanji-level reading accuracy of Japanese TTS systems. It was developed as part of the Sarashina2.2-TTS project, where it serves as the backbone of the Kana CER (Kana Character Error Rate) metric proposed in our paper.

Usage

With Transformers Pipeline

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
import torch

model_id = "sbintuitions/kana-whisper"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    use_safetensors=True,
).to("cuda")

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    torch_dtype=torch.float16,
    device="cuda",
    generate_kwargs={"language": "ja", "task": "transcribe"},
)

result = pipe("path/to/audio.wav")
print(result["text"])
# Example output: "キョーワイイテンキデスネ"

Citation

@misc{liu2026sarashina22ttstacklingkanjipolyphony,
      title={Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis}, 
      author={Lianbo Liu and Shiao Zhu and Kai Washizaki and Reo Yoneyama and Haesung Jeon and Mengjie Zhao and Yusuke Fujita and Hao Shi and Nao Yoshida and Yuan Gao and Roman Koshkin and Yukiya Hono and Yui Sudo},
      year={2026},
      eprint={2606.25369},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.25369}, 
}