qwen3-asr-0.6b-onnx-int4-merged

rhasspy/qwen3-asr-0.6b-onnx-int4 with its two decoder graphs replaced by a single merged decoder that accepts a KV cache alongside a dynamic sequence length.

This makes a context-biasing prompt cheap to reuse, and makes the package substantially smaller.

Why

Qwen3-ASR takes free-form text in the chat template's system turn to bias decoding toward particular spellings — the reason it can be told that "Ecobee" is a word. But the model must read that prompt before it starts decoding, at roughly 2.8 ms per token, so a 50-name vocabulary (~180 tokens) can double the time for a short command.

The prompt sits ahead of the audio, so its attention state depends only on the prompt tokens and never changes between utterances. The original split export cannot exploit that: decoder_init accepts no KV cache, and decoder_step is pinned to a single token. This export takes both a cache and a dynamic sequence length, so the prompt's state is computed once and reused.

Because the graph takes embeddings as input rather than doing the lookup internally, it also drops the in-graph embedding table: the shared decoder weights go from 962 MB to 340 MB.

Measurements

Raspberry Pi 5, 4 threads, 3.2 s command, 50-name prompt, compared against the split export:

The latency benefit scales with how much of the work is prompt. It is largest for short voice commands and small for long-form audio — a 30 s clip gained only ~1.04x. The memory saving goes the other way and grows with length: on that same 30 s clip, 4.16 GB → 2.87 GB.

Accuracy

Transcripts were compared against the split export directly: identical on 20/20 checks on x86, and 39/40 on a Pi 5 (10 short English clips × 4 prompt sizes, scored against ground truth). Both exports scored the same 39/40. The three differing transcripts were "13 percent" vs "thirteen percent" (both correct) and two spellings of a name that both exports get wrong when given no prompt.

This is not a WER evaluation. It is a check that the re-export did not change behavior on a small set of short English utterances. Independent quantization plus a restructured graph can flip near-tie tokens, so treat measured WER on the split export as indicative rather than transferred.

Requires a runtime that supports the merged layout

The graph filenames and input signature differ from the split export (decoder_merged.int4.onnx, with an attention_mask input; no decoder_init or decoder_step). Software that expects the split layout will not load this repo — use rhasspy/qwen3-asr-0.6b-onnx-int4 there instead, which remains published and unchanged.

In wyoming-faster-whisper, the layout is detected automatically from the files present:

wyoming-faster-whisper --stt-library qwen3-asr \
    --model rhasspy/qwen3-asr-0.6b-onnx-int4-merged \
    --language en --uri 'tcp://0.0.0.0:10300' --data-dir /data

Files

Provenance

The decoder was exported from Qwen/Qwen3-ASR-0.6B-hf with a DecoderMergedWrapper added to a clone of andrewleech/qwen3-asr-onnx, then quantized with the same recipe as the original decoders: RTN, block_size=64, accuracy_level=4. The encoder, tokenizer, config and embedding table are the files from rhasspy/qwen3-asr-0.6b-onnx-int4, byte for byte — the exported embedding table was verified identical to the published one before reuse.