Qwen3-ASR-0.6B-TW-Agent (joint S2T v6 + on-policy distillation)

LoRA adapter turning Qwen/Qwen3-ASR-0.6B into a Taiwan-localized speech-to-action agent (built for the office phone-attendant use case): it transcribes zh-TW/English speech and acts on it in the same decoder pass — no separate ASR + LLM pipeline.

The audio encoder is frozen; only decoder LoRA weights (r32) are trained, jointly on zh-TW ASR (Common Voice 17), English ASR (LibriSpeech train-clean-100), telephone-domain dialogs, and ~54k function-calling examples (50/50 zh/en, 13% "decline to call" negatives).

Output format: transcript-first

Every response to an audio turn starts with a verbatim transcript, then the action. The transcript is not decoration — the closing </transcript> gives the model a hard stop between hearing and acting (separately-trained adapters rambled without it), and it gives the calling application the raw ASR text for logging, confirmation prompts, or correction UIs.

1 — Caller's request maps to a tool (Hermes-format <tool_call>, one JSON object per call):

<transcript>請幫我轉接行銷部的陳怡君</transcript>
<tool_call>
{"name": "search_contacts", "arguments": {"query": "陳怡君", "department": "行銷"}}
</tool_call>

2 — No tool fits (trained to decline rather than force a call — see the irrelevance scores):

<transcript>你們公司幾點下班？</transcript>
我們的上班時間是週一至週五九點到六點。

3 — Text-only turns (tool results, follow-up questions) skip the transcript and reply or call tools directly, following the standard Qwen3 chat template with tools=[...].

Parsing: take everything between <transcript>...</transcript> as the ASR result, then scan for <tool_call>...</tool_call> blocks (JSON inside); anything else is the spoken reply.

Benchmarks (2026-07-30)

Tool-calling — BFCL v4, 11 non-agentic categories (no java/js), limit 150/cat, official ast_checker; TW-BFCL v4 is a zh-TW localization of the same categories:

ASR — 300 Common Voice 17 zh-TW test clips (CER, OpenCC s2tw-normalized) and 200 LibriSpeech test-clean clips (WER, case-insensitive), identical pipeline for all rows:

On-policy distillation fixed the English weakness

An earlier release of this adapter sat at 11.56 en WER — 3× worse than the base model — because fine-tuning on a Chinese-heavy mix caused catastrophic forgetting, and adding more English audio did not help (5k and 10k rows gave identical results). The cause was off-policy training: supervised fine-tuning on reference transcripts never visits the error states the model itself produces.

A round of on-policy distillation (arXiv 2605.28139) fixed it in 1500 steps: the student transcribes audio itself, a frozen Qwen3-ASR-1.7B teacher scores the student's own transcript on the same audio, and the loss is a temperature-scaled KL over the union of teacher/student top-k tokens. English WER went 11.56 → 2.33 (better than the base model, and within 0.08 of the teacher), at the cost of +0.18 zh CER and flat tool-calling.

Honest caveats:

• TW-BFCL numbers are not comparable to English-BFCL leaderboard numbers: its gold answers are augmented with zh alternatives (see the dataset card) and ~29% of AST rows are flagged _solvable=false (cross-lingual entity mismatch). On the _solvable subset this model scores 68.7.
• live_relevance (n=16) dropped to 75% after training the model to decline more readily.

Usage (speech → tool call)

from qwen_asr.core.transformers_backend.modeling_qwen3_asr import Qwen3ASRForConditionalGeneration
from qwen_asr.core.transformers_backend.processing_qwen3_asr import Qwen3ASRProcessor
from peft import PeftModel
import torch

base = "Qwen/Qwen3-ASR-0.6B"
proc = Qwen3ASRProcessor.from_pretrained(base)
thinker = Qwen3ASRForConditionalGeneration.from_pretrained(base, dtype=torch.bfloat16).thinker
model = PeftModel.from_pretrained(thinker, "<this repo>").cuda().eval()
# render your system prompt + tools + <|audio_pad|> user turn with the processor, then generate;
# the reply is <transcript>...</transcript> followed by a Hermes <tool_call> or a text answer.

For text-only use (e.g. BFCL harnesses), merge the adapter into a plain Qwen3ForCausalLM (the audio tower is not needed).

Trained with LoRA r32 on a frozen-encoder Qwen3-ASR-0.6B; ~81.5k-row multi-task mix, 1 epoch, lr 5e-5, bf16.