Qwen3-ASR-0.6B — GGUF (Jetson Nano gen1 optimized)

Single-file GGUF builds of Qwen/Qwen3-ASR-0.6B (audio encoder + Qwen3-0.6B LLM decoder) that run with CUDA on the original Jetson Nano gen1 (Tegra X1, sm_53, CUDA 10.2). Native Traditional Chinese + punctuation, zh/en code-switch.

Variants

Why q8_0, not q4_0, on the Jetson Nano (Maxwell sm_53)

Maxwell sm_53 has no dp4a (int8 dot-product), so ggml dequantizes low-bit weights and the extra ALU work makes q4_0 slower than q8_0/f16 on this GPU (measured ~1.7x slower). q4_0 also loses Traditional-character precision on this 0.6B model (output collapses to Simplified). So q8_0 is the sweet spot: half the size of f16, same speed and accuracy on Maxwell. q4_0 is only worth it on Ampere+ (fast low-bit GEMM) where Simplified output is acceptable.

Engines (both Jetson-Nano-gen1 optimized)

Two open engines run this GGUF on the Nano gen1; both share the cuFFT batched mel and the CUDA-10.2/sm_53 build fixes:

The standalone reaches lower RTF because it keeps encoder weights on the GPU and LLM weights on the CPU simultaneously (batch-1 decode is faster on the A57 than on Maxwell); RapidSpeech's unified single-weight-buffer keeps everything on GPU.

Usage

Standalone:

qwen3-asr-cli -m qwen3-asr-0.6b-q8_0.gguf -f clip_16k.wav -t 4

RapidSpeech:

rs-asr-offline -m qwen3-asr-0.6b-q8_0.gguf -w clip_16k.wav

Convert your own

From a local Qwen/Qwen3-ASR-0.6B checkpoint, using the RapidSpeech fork:

python scripts/convert_qwen3_asr_to_gguf.py --hf-dir /path/to/Qwen3-ASR-0.6B \
    --output qwen3-asr-0.6b-q8_0.gguf --quant q8_0