Qwen3-ASR-1.7B + LoRA — Cantonese (yue-HK) ASR, no-symbol target (PREVIEW)

LoRA adapter for Qwen/Qwen3-ASR-1.7B fine-tuned on the full official LegCo transcripts dataset (JackyHoCL/legco-yue-official-transcripts, 119,204 chunks / ~914 h, 396 meetings, all venues 2025-07..2026-10) with a no-symbol target: labels are the official AI realtime captions with every symbol removed except % - ( ) (kept for numbers / parentheticals). The model is trained to emit punctuation-free, space-separated transcripts.

Checkpoint selection (full validation / test)

Selected epoch-2 checkpoint gives 73% relative test-WER reduction vs zero-shot.

Data

• Source: JackyHoCL/legco-yue-official-transcripts (local copy of the published parquet, embedded 16 kHz mono PCM16 WAV; all venues — Chamber plenary + committee rooms)
• Label: sentence -> all Unicode P*/S* symbols and U+3000 replaced by a single space, whitespace collapsed; % - ( ) kept; rows with empty result or duration < 0.5 s dropped
• Splits are disjoint by meeting (source dataset guarantee)

Training

• LoRA on thinker LLM linear layers only (thinker.model.layers.*.self_attn.{q,k,v,o}_proj + thinker.model.layers.*.mlp.{gate,up,down}_proj), rank 16, alpha 32, dropout 0.05; audio tower frozen
• 4.0 epochs, lr 5e-05 cosine (warmup 5%), bf16, batch 4 x grad-accum 2, gradient checkpointing, ~19.4 h on one RTX 3080 20 GB
• Per-epoch on-the-fly eval during training: eval_loss on the full validation split + 96-row generation WER/CER on a seeded validation subset (TensorBoard); then every epoch checkpoint re-evaluated on full validation+test before selection (see table above)

Usage

Requires qwen-asr==0.0.6 + transformers==4.57.6 (the native transformers>=5 qwen3_asr classes do not match this checkpoint layout) and peft.

import torch
import soundfile as sf
from qwen_asr.core.transformers_backend import (
    Qwen3ASRForConditionalGeneration, Qwen3ASRProcessor)
from peft import PeftModel

processor = Qwen3ASRProcessor.from_pretrained("JackyHoCL/Qwen3-ASR-1.7B-cantonese-written-no-symbol-lora")
model = Qwen3ASRForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-ASR-1.7B", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "JackyHoCL/Qwen3-ASR-1.7B-cantonese-written-no-symbol-lora")
model = model.to("cuda").eval()

# The outer wrapper class has no `forward`; route it to thinker.forward
# (same patch as the official Qwen3-ASR fine-tuning script):
def _patch(outer):
    outer_cls = type(outer)
    if getattr(outer_cls.__dict__.get("forward", None), "_patched", False):
        return
    def forward(self, *args, **kwargs):
        return self.thinker.forward(*args, **kwargs)
    forward._patched = True
    outer_cls.forward = forward
_patch(model)

audio = sf.read("clip.wav", dtype="float32")[0]
if audio.ndim > 1:
    audio = audio.mean(axis=1)

# prefix: empty system + audio user turn, with forced Cantonese language tag
prefix = processor.apply_chat_template(
    [{"role": "system", "content": ""},
     {"role": "user", "content": [{"type": "audio", "audio": "placeholder"}]}],
    add_generation_prompt=True, tokenize=False)[0] + "language Cantonese<asr_text>"
inp = processor(text=[prefix], audio=[audio], return_tensors="pt").to("cuda")
out = model.generate(**inp, max_new_tokens=320)
new = out[0][inp["input_ids"].shape[1]:]
print(processor.batch_decode([new], skip_special_tokens=True)[0])

Usage — vLLM

Serve the base model with this adapter as a vLLM LoRA module (pip install "qwen-asr[vllm]"):

vllm serve Qwen/Qwen3-ASR-1.7B \
    --enable-lora --max-lora-rank 32 \
    --lora-modules no-symbol=JackyHoCL/Qwen3-ASR-1.7B-cantonese-written-no-symbol-lora \
    --max-model-len 4096 --gpu-memory-utilization 0.8 --port 8000

Call the OpenAI-compatible /v1/chat/completions endpoint with "model": "no-symbol" and an audio_url content item; the reply begins with the trained language Cantonese<asr_text> prefix — strip it for the plain transcript.

Notes

• Target format: language Cantonese<asr_text> + transcript (official Qwen3-ASR fine-tuning contract).
• no-symbol convention: output is expected to be free of punctuation (spaces separate phrases, as in the source captions); % - ( ) may appear.
• The audio tower (perceiver + conv stem) is frozen; LoRA covers LLM self-attention + MLP projections only.
• Eval WER/CER: Levenshtein over character/word tokens, normalized by reference length, punctuation stripped on both sides (metrics are therefore insensitive to the no-symbol convention itself).

Published 2026-10-13 14:06:31 — no_symbol preview run.