Qwen3-ASR-1.7B · NInfer

简体中文

Model overview

A BF16 NInfer version of Qwen3-ASR-1.7B-hf, supporting automatic language identification and transcription in 30 languages and 22 Chinese dialects. It runs with NInfer Extended, with native C++/CUDA execution for audio encoding, prefill and decode.

This repository also provides a 0.6B draft model for faster inference and Qwen3-ForcedAligner-0.6B for word timestamps. The main model accepts batches of up to four audio files, automatic or specified language, context prompts and hotwords.

Performance

Audio s/s is the number of audio seconds processed per inference second: total input duration ÷ inference time. Results are warm-run BF16 medians, including audio encoding, prefill, complete transcription and synchronization, excluding model loading and CPU feature extraction. Transformers uses compiled prefill and CUDA Graph for decode.

Single 1.7B model

The 60-second samples repeat a fixed recording:

Four-audio batches

Prefill and decode

Prefill includes audio encoding, language processing and the first token. Decode starts at the second token and includes EOS; batch values are aggregate tok/s.

0.6B draft acceleration

The 1.7B target runs with a 0.6B draft: K=5 on RTX 5070 Ti and K=3 on RTX 6000D.

Complete prefill and decode with drafting enabled:

A 60-second mixed recording made from seven distinct clips and silence intervals:

Four-lane MTP

RTX 5070 Ti, BF16 1.7B target + 0.6B draft, K=5. Six warm-run medians, including both models' audio encoding, prefill and complete decode; loading and CPU feature extraction are excluded.

Engine 4a527133 · Four-lane measurements.

ForcedAligner latency

BF16, one request, including audio encoding, language processing and timestamp classification. The Transformers reference uses eager execution; NInfer uses CUDA Graph.

Accuracy

73 human-labelled English recordings, totalling 481.03 seconds and 1,150 reference words. WER and CER are word and character error rates; lower is better. WER uses normalized words; CER excludes spaces.

Agreement with Transformers

Draft decoding agreement

Complete output tokens compared with the single 1.7B model on the same 73 recordings.

Timestamp agreement

400 time boundaries across four recordings, compared with the official ForcedAligner reference.

Usage

Build

Requirements: Linux or WSL2, Python 3.11, CUDA with sm_120a support, cuDNN 9 and cuBLAS. ASR and alignment share one Python environment. See the build guide for dependencies.

git clone https://github.com/ByronLeeeee/ninfer-extended.git
cd ninfer-extended
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCUDNN_ROOT=/path/to/cudnn -DCUBLAS_ROOT=/path/to/cublas
cmake --build build --target ninfer-asr ninfer-align -j
pip install -r tools/qwen3_asr/requirements.txt

Use main on RTX 5070 Ti. The RTX 6000D K=3 results use revision 64395f47; to reproduce this configuration, run git checkout 64395f477fa8e5134f11edc5eb97697c47656d82 before building.

Download and transcribe

pip install huggingface_hub
hf download ByronLeeee/Qwen3-ASR-1.7B-hf-Ninfer \
  qwen3-asr-1.7b-bf16.ninfer --local-dir models
ffmpeg -i recording.wav -ac 1 -ar 16000 -c:a pcm_s16le recording-16k.wav
python tools/qwen3_asr/transcribe.py \
  --artifact models/qwen3-asr-1.7b-bf16.ninfer \
  --engine build/apps/ninfer-asr --audio recording-16k.wav \
  --warmups 0 --repeats 1 --out transcription.json

transcription.json contains text and detected language. Pass 1–4 files to --audio. Language detection is automatic; optional controls include --language English, --prompt "This is a medical lecture" and --hotwords aspirin hemoglobin.

Enable draft acceleration

hf download ByronLeeee/Qwen3-ASR-1.7B-hf-Ninfer \
  qwen3-asr-0.6b-bf16.ninfer --local-dir models
python tools/qwen3_asr/transcribe.py \
  --artifact models/qwen3-asr-1.7b-bf16.ninfer --engine build/apps/ninfer-asr \
  --draft-artifact models/qwen3-asr-0.6b-bf16.ninfer --draft-tokens 5 \
  --audio recording-16k.wav --warmups 0 --repeats 1 --out transcription.json

On RTX 5070 Ti, use --draft-tokens 5 and pass 1–4 WAV files to --audio. The historical single-lane RTX 6000D recipe uses --draft-tokens 3.

Add word timestamps

Transcribe first, then align the resulting text:

hf download ByronLeeee/Qwen3-ASR-1.7B-hf-Ninfer \
  qwen3-forced-aligner-0.6b-bf16.ninfer --local-dir models
TRANSCRIPT="$(python -c 'import json; print(json.load(open("transcription.json", encoding="utf-8"))["runs"][-1]["text"][0])')"
LANGUAGE="$(python -c 'import json; print(json.load(open("transcription.json", encoding="utf-8"))["runs"][-1]["language"][0])')"
python tools/qwen3_forced_aligner/align.py \
  --artifact models/qwen3-forced-aligner-0.6b-bf16.ninfer \
  --engine build/apps/ninfer-align --audio recording-16k.wav \
  --text "$TRANSCRIPT" --language "$LANGUAGE" \
  --warmups 0 --repeats 1 --out timestamps.json

Word results are in cases[0].runs[-1].words[0] of timestamps.json, with text, start and end fields in seconds. Edited transcripts can be passed directly to --text. Alignment supports Chinese, English, Cantonese, French, German, Italian, Japanese, Korean, Portuguese, Russian and Spanish.

ASR guide · ForcedAligner guide

Model files

All models use BF16 and are licensed under Apache-2.0.

Full results · Draft results · Alignment results · Artifact manifest