This repository provides research checkpoints for Mandarin target-speaker automatic speech recognition (TS-ASR) in multi-talker speech. The models identify and transcribe a designated speaker using one or both of the following cues:
The repository includes text-guided, enrollment-guided, and dual-cue Conformer-CTC variants. The dual-cue models combine complementary lexical and speaker information within a shared recognition architecture.
| Property | Value |
|---|---|
| Language | Mandarin Chinese |
| Task | Target-speaker ASR in two-speaker mixtures |
| Acoustic backbone | Conformer |
| Training objective | CTC |
| Supported guidance | Text cue, enrollment speech, or both |
| Audio sampling rate | 16 kHz |
| Acoustic features | 80-bin filterbanks, 25 ms frame length, 10 ms frame shift |
| Text-cue length used by the configs | 2-5 Chinese characters |
| Tokenization | Character tokenizer with 8,408 entries |
| Evaluation dataset | Unified TS-ASR Evaluation Set |
Unified-TS-ASR/
└── ckpt/
├── config/
│ ├── seg_dict_ext
│ └── tokens_kws8408.json
├── unified-ts-asr-concat/
├── unified-ts-asr-parallel/
├── text-cue-ts-asr/
├── text-cue-ts-asr-neg-0p3/
├── text-cue-ts-asr-neg-0p3-sv-head/
├── text-cue-ts-asr-neg-0p3-sv-head-0p1-reg/
├── text-cue-ts-asr-neg-0p3-sv-head-0p5-reg-0p01/
├── text-cue-ts-asr-neg-0p5/
├── speaker-cue-ts-asr-embedding-free/
├── speaker-cue-ts-asr-embedding-free-low-frame/
└── speaker-cue-ts-asr-embedding/
Each model directory contains:
config.yaml
model.pt
model.pt contains the corresponding DeepSpeed model state. The matching config.yaml, shared token list, and segmentation dictionary are required to reconstruct each model.
| Checkpoint | Cue support | Main distinction |
|---|---|---|
unified-ts-asr-concat |
Text + enrollment | Concatenates the two cue representations before conditioning the Conformer encoder |
unified-ts-asr-parallel |
Text + enrollment | Processes the two cue paths with parallel fusion |
Both dual-cue configurations enable use_text_cue and use_enroll_cue. Their evaluation interface can select text-only, enrollment-only, both-cue, or no-cue conditions when used with the matching project implementation.
| Checkpoint | Configuration distinction |
|---|---|
text-cue-ts-asr |
Text-cue baseline with positive-cue sampling |
text-cue-ts-asr-neg-0p3 |
Uses 30% negative-cue sampling (positive_prob: 0.7) |
text-cue-ts-asr-neg-0p5 |
Uses 50% negative-cue sampling (positive_prob: 0.5) |
text-cue-ts-asr-neg-0p3-sv-head |
Adds an auxiliary speaker-classification head with weight 0.1 |
text-cue-ts-asr-neg-0p3-sv-head-0p1-reg |
Speaker-head weight 0.1 with weight regularization 0.1 |
text-cue-ts-asr-neg-0p3-sv-head-0p5-reg-0p01 |
Speaker-head weight 0.5 with weight regularization 0.01 |
The text cue is encoded with KeywordTransformerEncoderDae and injected into ConformerKceEncoderDae through cross-attention.
| Checkpoint | Enrollment conditioning |
|---|---|
speaker-cue-ts-asr-embedding-free |
Direct enrollment-feature cross-attention without a separate pretrained speaker encoder |
speaker-cue-ts-asr-embedding-free-low-frame |
Embedding-free variant with encoded enrollment-frame reduction (enroll_lfr_m: 3, enroll_lfr_n: 3) |
speaker-cue-ts-asr-embedding |
Online CAM++ speaker embeddings with FiLM conditioning |
The CAM++ variant expects a frozen 192-dimensional Chinese speaker-verification encoder. Its configured initialization path is project-local and must be supplied separately when reconstructing that model.
Download the complete repository from the anonymous model page using its Download ZIP action. Do not use or cite a non-anonymous mirror during double-blind review.
These are custom FunASR-style research models rather than Transformers AutoModel checkpoints. They cannot be loaded directly with AutoModel.from_pretrained().
For each experiment, keep the configuration and checkpoint paired:
CONFIG=ckpt/<model-name>/config.yaml
CHECKPOINT=ckpt/<model-name>/model.pt
TOKEN_LIST=ckpt/config/tokens_kws8408.json
SEG_DICT=ckpt/config/seg_dict_ext
Reconstruction requires the custom classes named in the selected configuration, including the corresponding model, cue encoder, Conformer encoder, frontend, tokenizer, and dataset implementation. The implementation is available from the anonymous code repository. Several training-only paths in the YAML files—such as local manifests, DeepSpeed settings, and the CAM++ initialization path—refer to the original project environment and should be replaced for a new environment.
The paired evaluation dataset supplies cue_text, enroll_source, target transcriptions, speaker identifiers, SIR, and overlap metadata. See the anonymous dataset card for its directory layout and JSONL schemas.
The intended benchmark contains five evaluation domains:
aishell1-tsaishell-ios-tsaishell-android-tsaishell-mic-tsaishell-mix-tsFor text-guided evaluation, use the fixed cue2, cue3, cue4, and cue5 metadata files instead of sampling cues at evaluation time. For enrollment and dual-cue evaluation, use the corresponding *-spkemb.jsonl files.
Recommended reporting includes:
No benchmark numbers are included in this model card because results depend on the matching inference implementation and scoring configuration.
These checkpoints are intended for research on Mandarin target-speaker ASR, heterogeneous target cues, robustness to interfering speakers, cue ablations, and device or domain shift.
They are not intended as a general-purpose speech recognizer or as a production-ready speaker-identification system. The models were designed around two-speaker mixtures and controlled Mandarin evaluation data; performance on other languages, unconstrained real-world conversations, more than two simultaneous speakers, or unseen acoustic conditions requires separate validation.
The repository declares the Apache 2.0 license. Users must also comply with the terms of the datasets and pretrained components used with these checkpoints.
ckpt/unified-ts-asr-parallel/model.pt 159 MBckpt/text-cue-ts-asr-neg-0p3-sv-head-0p1-reg/model.pt 151 MBckpt/text-cue-ts-asr-neg-0p3-sv-head-0p5-reg-0p01/model.pt 151 MBckpt/text-cue-ts-asr-neg-0p3-sv-head/model.pt 151 MBckpt/unified-ts-asr-concat/model.pt 149 MBckpt/text-cue-ts-asr/model.pt 149 MBckpt/text-cue-ts-asr-neg-0p3/model.pt 149 MBckpt/text-cue-ts-asr-neg-0p5/model.pt 149 MBckpt/speaker-cue-ts-asr-embedding/model.pt 141 MBckpt/speaker-cue-ts-asr-embedding-free-low-frame/model.pt 120 MBckpt/speaker-cue-ts-asr-embedding-free/model.pt 119 MB