Qwen3-ASR-1.7B + LoRA — Cantonese (yue-HK) ASR (v4) - Written Language

A LoRA adapter for Qwen/Qwen3-ASR-1.7B (the official Qwen3-ASR audio-text LLM) that substantially improves Cantonese (yue-HK) speech recognition. It is fine-tuned on ~207 h of cleaned LegCo (Hong Kong Legislative Council) webcasts with official-record transcripts (written_yue) as targets. The audio tower stays frozen; LoRA (rank 16) covers the LLM self-attention and MLP projections — 17.4 M trainable parameters (0.85% of 2.06 B), so the adapter is a single ~70 MB file on top of the base model.

The adapter is intended for Cantonese ASR on council/debate-style speech: 12+ speakers per meeting, code-mixed English, numbers, dates, and Cantonese-informal-but-standard register. Expected output convention (official Qwen3-ASR contract): language Cantonese<asr_text> + transcript; force text-only output by prefixing the language tag.

Results

Held-out splits, WER/CER (punctuation & spaces stripped, character tokens):

Checkpoint selection — full-split eval of every epoch (3-epoch run):

Epoch 2 was selected by full test+dev WER (the per-epoch 48-row TensorBoard subset had suggested epoch 3).

Data

• Source: JackyHoCL/legco-yue-asr-preview v4 snapshot (57,566 utterances / 227.9 h / 56 sessions, 16 kHz mono WAV, sliced ≤20 s)
• Training label: written_yue (official-record written Cantonese)
• Filters (all splits): drop rows with any QC flag and match_confidence < 0.5 → 50,937 rows (train 39,134 / dev 5,891 / test 5,912, ~207 h)

Training

• LoRA on thinker.model.layers.*.self_attn.{q,k,v,o}_proj + thinker.model.layers.*.mlp.{gate,up,down}_proj — rank 16, alpha 32, dropout 0.05; audio tower frozen
• 3 epochs, lr 5e-5 cosine (warmup 5%), bf16, batch 4 × grad-accum 4, gradient checkpointing, 6 h 20 m on one RTX 3080 20 GB
• Target: language Cantonese<asr_text> + transcript (official Qwen3-ASR fine-tuning contract)

Usage — Transformers (this adapter)

Requires qwen-asr==0.0.6 + transformers==4.57.6 (the native transformers>=5 qwen3_asr classes do not match this checkpoint layout) and peft.

import torch
import soundfile as sf
from qwen_asr.core.transformers_backend import (
    Qwen3ASRForConditionalGeneration, Qwen3ASRProcessor)
from peft import PeftModel

processor = Qwen3ASRProcessor.from_pretrained("Qwen/Qwen3-ASR-1.7B")
model = Qwen3ASRForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-ASR-1.7B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "JackyHoCL/qwen3-asr-yue-cantonese-lora")
model = model.to("cuda").eval()

# The outer wrapper class has no `forward`; route it to thinker.forward
# (same patch as the official Qwen3-ASR fine-tuning script):
def _patch(outer):
    outer_cls = type(outer)
    if getattr(outer_cls.__dict__.get("forward", None), "_patched", False):
        return
    def forward(self, *args, **kwargs):
        return self.thinker.forward(*args, **kwargs)
    forward._patched = True
    outer_cls.forward = forward
_patch(model)

audio = sf.read("clip.wav", dtype="float32")[0]
if audio.ndim > 1:
    audio = audio.mean(axis=1)

# prefix: empty system + audio user turn, with forced Cantonese language tag
prefix = processor.apply_chat_template(
    [{"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": "placeholder"}]}],
    add_generation_prompt=True, tokenize=False)[0] + "language Cantonese<asr_text>"
inp = processor(text=[prefix], audio=[audio], return_tensors="pt").to("cuda")
out = model.generate(**inp, max_new_tokens=512)
new = out[0][inp["input_ids"].shape[1]:]
print(processor.batch_decode([new], skip_special_tokens=True)[0])

Usage — vLLM

Official stack: pip install "qwen-asr[vllm]" (extra pins a vLLM release; newer vLLM such as 0.30 also works with the bundled qwen_asr.core.vllm_backend model implementation).

Serving this LoRA adapter (recommended)

qwen-asr-serve is the official wrapper around vllm serve with the Qwen3-ASR model registry pre-loaded. Serve the base model with the adapter as a vLLM LoRA module (rank 16 → --max-lora-rank 32):

vllm serve Qwen/Qwen3-ASR-1.7B \
    --enable-lora --max-lora-rank 32 \
    --lora-modules qwen3-asr-yue-cantonese=JackyHoCL/Qwen3-ASR-1.7B-cantonese-written-lora \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.8 \
    --port 8000

Clients choose the adapter or the base by model name:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-asr-yue-cantonese",
    "messages": [
      {"role": "system", "content": ""},
      {"role": "user", "content": [
        {"type": "audio_url",
         "audio_url": {"url": "https://example.com/clip.wav"}}
      ]}
    ],
    "max_tokens": 512,
    "temperature": 0
  }'

The free-form response starts with the trained language Cantonese<asr_text> prefix — strip it to get the plain transcript (e.g. qwen_asr.parse_asr_output or a regex on the leading language …<asr_text> run).

Notes

• Repo role: release builds. Dev / test / preview: JackyHoCL/qwen3-asr-yue-lora-preview.
• The audio tower (perceiver + conv stem) is frozen; LoRA covers LLM self-attention + MLP projections only.
• Eval WER/CER: Levenshtein over character tokens, normalized by reference length, punctuation stripped on both sides.
• Lineage: v1 (8 h) → v2 (110 h) → v3 (103.7 h refreshed) → v4 (227.9 h, 3 epochs, best-of-3 checkpoint).