Qwen3-ASR-0.6B + LoRA — Cantonese (yue-HK) ASR — Written Language

A LoRA adapter for Qwen/Qwen3-ASR-0.6B (the small official Qwen3-ASR audio-text LLM) that substantially improves Cantonese (yue-HK) speech recognition. It is fine-tuned on ~207 h of cleaned LegCo (Hong Kong Legislative Council) webcasts with official-record transcripts (written_yue) as targets — the same v4 data and recipe as the 1.7B release adapter, so results are directly comparable. The audio tower stays frozen; LoRA (rank 16) covers the LLM self-attention and MLP projections — 10.1 M trainable parameters (1.27% of 792 M), so the adapter is a single ~40 MB file on top of the base model.

The adapter is intended for Cantonese ASR on council/debate-style speech: 12+ speakers per meeting, code-mixed English, numbers, dates, and Cantonese-informal-but-standard register. Expected output convention (official Qwen3-ASR contract): language Cantonese<asr_text> + transcript; force text-only output by prefixing the language tag.

Results

Held-out full splits, WER/CER (punctuation & spaces stripped):

Checkpoint selection — full test+dev WER of every epoch: the per-epoch 48-row TensorBoard subset had suggested epoch 3, but the complete splits select epoch 2 (0.191 test / 0.178 dev) over epoch 3 (0.197 / 0.179), matching the 1.7B v4 selection procedure. Relative to the zero-shot 0.6B base: −59 % WER on test. Reference on the same test set: 1.7B v4 adapter 0.174 vs its zero-shot base 0.326.

Data

• Source: JackyHoCL/legco-yue-asr-preview v4 snapshot (57,566 utterances / 227.9 h / 56 sessions, 16 kHz mono WAV, sliced ≤20 s)
• Training label: written_yue (official-record written Cantonese)
• Filters (all splits): drop rows with any QC flag and match_confidence < 0.5 → 50,937 rows (train 39,134 / dev 5,891 / test 5,912, ~207 h)

Training

• LoRA on thinker.model.layers.*.self_attn.{q,k,v,o}_proj + thinker.model.layers.*.mlp.{gate,up,down}_proj — rank 16, alpha 32, dropout 0.05; audio tower frozen
• 3 epochs, 7,338 steps, lr 5e-5 cosine (warmup 5%), bf16, batch 16 (per-device, no grad-accum → global 16), gradient checkpointing, ~3 h 45 m on one RTX 3080 20 GB
• Target: language Cantonese<asr_text> + transcript (official Qwen3-ASR fine-tuning contract); seed 42

Usage — Transformers (this adapter)

Requires qwen-asr==0.0.6 + transformers==4.57.6 (the native transformers>=5 qwen3_asr classes do not match this checkpoint layout) and peft.

import torch
import soundfile as sf
from qwen_asr.core.transformers_backend import (
    Qwen3ASRForConditionalGeneration, Qwen3ASRProcessor)
from peft import PeftModel

processor = Qwen3ASRProcessor.from_pretrained("Qwen/Qwen3-ASR-0.6B")
model = Qwen3ASRForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-ASR-0.6B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "JackyHoCL/Qwen3-ASR-0.6B-cantonese-written-lora")
model = model.to("cuda").eval()

# The outer wrapper class has no `forward`; route it to thinker.forward
# (same patch as the official Qwen3-ASR fine-tuning script):
def _patch(outer):
    outer_cls = type(outer)
    if getattr(outer_cls.__dict__.get("forward", None), "_patched", False):
        return
    def forward(self, *args, **kwargs):
        return self.thinker.forward(*args, **kwargs)
    forward._patched = True
    outer_cls.forward = forward
_patch(model)

audio = sf.read("clip.wav", dtype="float32")[0]
if audio.ndim > 1:
    audio = audio.mean(axis=1)

# prefix: empty system + audio user turn, with forced Cantonese language tag
prefix = processor.apply_chat_template(
    [{"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": "placeholder"}]}],
    add_generation_prompt=True, tokenize=False)[0] + "language Cantonese<asr_text>"
inp = processor(text=[prefix], audio=[audio], return_tensors="pt").to("cuda")
out = model.generate(**inp, max_new_tokens=512)
new = out[0][inp["input_ids"].shape[1]:]
print(processor.batch_decode([new], skip_special_tokens=True)[0])

Usage — vLLM

Official stack: pip install "qwen-asr[vllm]" (extra pins a vLLM release; newer vLLM such as 0.30 also works with the bundled qwen_asr.core.vllm_backend model implementation).

Serving this LoRA adapter (recommended)

Serve the base model with the adapter as a vLLM LoRA module (rank 16 → --max-lora-rank 32):

vllm serve Qwen/Qwen3-ASR-0.6B \
    --enable-lora --max-lora-rank 32 \
    --lora-modules qwen3-asr-06b-cantonese-written=JackyHoCL/Qwen3-ASR-0.6B-cantonese-written-lora \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.8 \
    --port 8000

Clients choose the adapter or the base by model name:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-asr-06b-cantonese-written",
    "messages": [
      {"role": "system", "content": ""},
      {"role": "user", "content": [
        {"type": "audio_url",
         "audio_url": {"url": "https://example.com/clip.wav"}}
      ]}
    ],
    "max_tokens": 512,
    "temperature": 0
  }'

The free-form response starts with the trained language Cantonese<asr_text> prefix — strip it to get the plain transcript (e.g. qwen_asr.parse_asr_output or a regex on the leading language …<asr_text> run).

Notes

• Repo role: 0.6B release build. Siblings: JackyHoCL/qwen3-asr-yue-cantonese-lora (1.7B release) and JackyHoCL/qwen3-asr-yue-lora-preview (dev / test / preview).
• The audio tower (perceiver + conv stem) is frozen; LoRA covers LLM self-attention + MLP projections only.
• Eval WER/CER: Levenshtein over character tokens, normalized by reference length, punctuation stripped on both sides.
• This adapter was trained from scratch on the v4 snapshot (3 epochs). A continued run on the extended 281.8 h / 69-session snapshot (2× RTX
  3080. is in progress and may supersede these weights.