Qwen3-ASR text decoders, fused for onnxruntime-web WebGPU
The text decoder of Qwen/Qwen3-ASR-0.6B and
Qwen/Qwen3-ASR-1.7B, re-exported with the
onnxruntime-genai model builder for
ORT's WebGPU execution provider. Meant as a drop-in replacement for the split
decoder_init / decoder_step graphs of
andrewleech/qwen3-asr-{0.6b,1.7b}-onnx,
whose encoder.int4.onnx, embed_tokens.bin and tokenizer.json are still needed.
One graph serves prefill and step:
GroupQueryAttentionwith RoPE folded in (Q/K RMSNorm kept as explicit nodes so the graph loads on onnxruntime-web 1.27's GQA schema); fused QKV and gate/up projections;SkipSimplifiedLayerNormalization.- int4
MatMulNBits, RTN, block size 32, accuracy level 4 — the layout ORT's DP4A and subgroup-matrix kernels require. - lm_head computes the last position only (
prune_lm_head), so prefill returnslogits [batch, 1, vocab]and never reads back a[seq, vocab]tensor. - 519 nodes per token (the split export ran 1586).
| File | I/O precision | Sidecar |
|---|---|---|
decoder-0.6b-fp16.onnx |
fp16 | 352 MB |
decoder-0.6b-fp32.onnx |
fp32 | 406 MB |
decoder-1.7b-fp16.onnx |
fp16 | 985 MB |
decoder-1.7b-fp32.onnx |
fp32 | 1.11 GB |
Same weights at both precisions; the fp32 graphs are for adapters without shader-f16.
Each proto names its sidecar as <name>.onnx.data, so all pairs can sit in one directory.
Interface
Inputs: inputs_embeds [B, S, H] (H = 1024 / 2048; fp16 rows from embed_tokens.bin,
with the encoder's audio_features written over the <|audio_pad|> positions),
attention_mask [B, total] int64 (its row sum is the real sequence length), and
past_key_values.{0..27}.{key,value} [B, 8, past, 128] (empty, past = 0, at prefill).
Outputs: logits [B, 1, 151936], present.{0..27}.{key,value}.
Prompt ids, mel parameters and the language header (language X<asr_text>) are as in the
andrewleech export. The header can be appended to the prompt to pin the language.
Measured (Apple M1 Pro, Chrome, onnxruntime-web 1.27 WebGPU, 30 s chunk)
| Build | prefill | per step | 30 s clip |
|---|---|---|---|
| 1.7B fp16 (this repo) | 0.73 s | 19.5 ms | 3.4 s |
| 1.7B split export, block 64 | 5.5–6.5 s | 55–60 ms | ~15 s |
| 0.6B fp16 (this repo) | 0.25 s | 14.3 ms | 2.3 s |
| 0.6B split export, block 64 | — | 47.6 ms | — |
Greedy output on LibriSpeech test-other matches the split int4 export up to quantization-level differences (a hyphen, a comma).
Recipe
# extract thinker.model.* + thinker.lm_head.weight into a Qwen3ForCausalLM dir, then:
python -m onnxruntime_genai.models.builder -i <dir> -o out -p int4 -e webgpu \
--extra_options exclude_embeds=true prune_lm_head=true block_size=32 accuracy_level=4 \
enable_webgpu_graph=true fuse_qk_norm_gqa=false [use_webgpu_fp32=true]