DuplexJev-B-Whisper-small-SmolLM3-3B

A small DuplexJev connector: the Whisper-small encoder (frozen) feeds a frozen SmolLM3-3B, and every typed question is read as a single-token distribution over its options, with zero decode steps and many questions per forward pass. Trained with the same recipe as the Qwen3-32B connectors released with the paper.

Quick start

pip install "duplexjev[speech]>=0.2.2"

from duplexjev import Decider, Question

d = Decider.from_pretrained("adventists-ai/DuplexJev-B-Whisper-small-SmolLM3-3B")

d.decide("utterance_en.wav", [
    Question("turn", "Has the user finished speaking?", ["finished", "not finished"], lang="en"),
    Question("intent", "What does the user want?", ["climate", "media", "navigation", "phone", "chit-chat"], lang="en"),
], lang="en")
d.decide("utterance_zh.wav", [                            # Chinese speech: Chinese question and options
    Question("turn", "用户说完了吗？", ["说完了", "还没说完"], lang="zh"),
    Question("intent", "用户想做什么？", ["空调", "媒体", "导航", "电话", "闲聊"], lang="zh"),
], lang="zh")

The connector was trained with SmolLM3's no-think format (an empty <think></think> block before the answer). The tokenizer in this repository carries that chat template; SmolLM3's default template adds a long system prompt with today's date and would not match.

Results

Leaderboard scores (see the GitHub README): main language (mean of qa100, ZJU-ML, Easy-Turn) 37.4, paralinguistics (mean of gender, emotion) 41.5.

Paper protocol: the evaluation prompts of the paper (single option order, one H200). duplexjev defaults: the package's own prompts with lang matched to the clip. Spoken QA with a small LLM is bounded by the LLM itself (see the transcript-reading row); use these connectors for short decisions and speaker cues rather than open knowledge questions.

Latency of one decision event (10 questions, one 4.5 s clip, duplexjev 0.2.2, plain PyTorch): 6.5 s on 8 CPU threads (Xeon Platinum 8558, fp32, not quantized); about 149 ms on one H200 (bf16, measured while the GPU was shared with a training job).

Training

Frozen encoder and LLM; only the connector is trained: R1 and R2 transcript distillation on two disjoint 0.5 M-utterance packs of the Ultravox v0.6 mixture (32k steps each, global batch 16). Code and scripts: https://github.com/adventists-ai/duplexjev.

Limitations

• Evaluated on read or acted speech and small test sets (100–800 items); not on streaming input.
• Small LLMs answer knowledge questions poorly even from the transcript.
• Emotion labels come from acted corpora; do not use the outputs to make decisions about individuals.
• The connector only works with the encoder and LLM listed above.

License

Apache-2.0 for the connector weights. Some training corpora (e.g. WenetSpeech, CoVoST 2) are licensed for non-commercial use only; check them before commercial use. The frozen models keep their own licences: SmolLM3-3B (Apache-2.0) and the encoder (Apache-2.0).

Citation

@inproceedings{jin2027duplexjev,
  title     = {Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen {LLM} Hear Beyond the Transcript},
  author    = {Jin, Jie and Ma, Ziyin and Yin, Min and Chen, Jinyu and Song, Haigang and Pang, Zhikun and Zhang, Xiaowen},
  booktitle = {Submitted to IEEE ICASSP},
  year      = {2027}
}

Built by Adventists.ai. Claude (Anthropic) assisted with code.