orukqwen-asr-1.7b · v2
Eight RL runs averaged into one speech-recognition model. v2 continues our RL post-training of Qwen3-ASR-1.7B [1] with a second, style-pinned GSPO stage, averages eight independent runs into one model, and ships with a beam-search decoding config.
| Result | What it measures |
|---|---|
| 4.66 mean WER | 9 public English sets, official harness (v1: 4.76 · base: 4.94) |
| 5.42 mean WER | equal-weight mean over 7 public English test sets in our November 7 official-harness run; see the dated comparison below |
| −0.17* WER | vs v1's weights on 11,080 clips from 5 corpora used nowhere in training or tuning (−0.78* on AR-glasses far-field audio) |
| 2.04B parameters | same architecture, tokenizer and prompt as Qwen/Qwen3-ASR-1.7B-hf; the default decoding configuration changes from v1 |
Authors: Nathan Roll, Vianney Grenez, Busra Marsan and Irene Yi (oruk).
The model's 1.7B name follows the Qwen model family; its complete parameter count is 2.04B. It is the same architecture as the base model, rather than an ensemble at inference.
Dated public-set comparison
| Model | Mean WER over the same 7 public sets (%) | Source |
|---|---|---|
| OrukQwen 2 | 5.4219 | Our run of the official harness on November 7, 2026 with the shipped 3-beam decoding configuration |
| NVIDIA Nemotron-3.5-Transcribe-30B-A4B | 5.5114 | Equal-weight mean of displayed board columns read November 9, 2026 |
| Qwen3-ASR-1.7B-hf | 5.7457 | Equal-weight mean of displayed board columns read November 9, 2026 |
The seven sets are AMI-Cleaned, Earnings22-Cleaned-AA-chunked, Gigaspeech-Cleaned, SPGISpeech, URGENT2024, Voice Arena Monsoon and Voxpopuli-AA-Cleaned. These are equal-weight means of dataset WERs rather than pooled word counts. Comparator columns are rounded by the board; the extra digits above describe their arithmetic mean rather than extra measurement precision. Our own base-model run is 5.747% on these sets.
This is a dated public-set comparison and not an official leaderboard rank. The board's ranking includes two private test-set aggregates that its maintainers run. We do not claim that the maintainers have evaluated or accepted this release. The board has changed since the comparison date; this card makes no current “#1” claim.
What's new in v2
- Better weights: style-pinned RL. Starting from v1, we ran eight more GSPO runs with a reward that scores content against the human reference while pinning transcription style (fillers, disfluent repeats, acronym spacing) to v1's own transcript of the same clip. v9 of our training campaign found that ordinary RL gains on ASR are mostly style drift: the model learns to drop or keep repeats and fillers, which wins on one benchmark and loses on another. Pinning the style lets the content signal through. Every member of the chosen eight-run pool is retained with equal weight. The decision to combine its two pin-strength groups followed development results; see Evaluation integrity.
- Better decoding.
generation_config.jsonnow uses 3-beam search, a no-repeat-8-gram guard against loops, and a ban on CJK tokens, so English output cannot drift into Chinese/Japanese/Korean script.
Each part helps on its own, and they stack:
Results
Open ASR Leaderboard, official harness
Official harness (huggingface/open_asr_leaderboard @ edd684247365, transformers/run_eval.py unchanged, repo normalizer and scorer), bf16, language="en", 1× A100-SXM4-40GB. Measured once, on 2026-11-07. WER %, lower is better. Δ columns are paired per-utterance bootstraps (2,000 resamples); * = the 95% interval excludes zero. The best value in each row is in bold.
| Set | Base Qwen3-ASR-1.7B | v1 (greedy) | v1 weights + v2 decoding | v2 | Δ v2 vs v1 weights at the same decoding | Δ v2 vs v1 | Δ v2 vs base |
|---|---|---|---|---|---|---|---|
| AMI-Cleaned | 8.32 | 7.92 | 7.96 | 7.68 | -0.28 [-0.37, -0.18]* | -0.24 [-0.32, -0.15]* | -0.64 [-0.75, -0.51]* |
| Earnings22-Cleaned-AA-chunked | 5.84 | 5.85 | 5.74 | 5.89 | +0.16 [-0.07, +0.49] | +0.04 [-0.39, +0.40] | +0.05 [-0.37, +0.65] |
| Gigaspeech-Cleaned | 7.22 | 7.11 | 7.08 | 7.10 | +0.02 [-0.01, +0.04] | -0.01 [-0.04, +0.01] | -0.12 [-0.15, -0.09]* |
| LibriSpeech test-clean | 1.25 | 1.20 | 1.17 | 1.19 | +0.02 [-0.01, +0.06] | 0.00 [-0.04, +0.04] | -0.06 [-0.10, -0.01]* |
| LibriSpeech test-other | 2.94 | 2.89 | 2.81 | 2.80 | -0.01 [-0.07, +0.05] | -0.09 [-0.16, -0.03]* | -0.15 [-0.23, -0.07]* |
| SPGISpeech | 2.58 | 2.75 | 2.73 | 2.75 | +0.03 [+0.01, +0.04]* | +0.01 [0.00, +0.02] | +0.18 [+0.16, +0.20]* |
| URGENT2024 | 9.93 | 9.48 | 9.37 | 9.17 | -0.19 [-0.41, +0.03] | -0.31 [-0.55, -0.05]* | -0.76 [-1.11, -0.42]* |
| Voice Arena Monsoon | 3.48 | 3.50 | 3.51 | 3.50 | -0.01 [-0.08, +0.06] | 0.00 [-0.08, +0.07] | +0.02 [-0.07, +0.11] |
| Voxpopuli-AA-Cleaned | 2.86 | 2.11 | 2.01 | 1.86 | -0.15 [-0.28, -0.03]* | -0.25 [-0.39, -0.12]* | -1.01 [-1.22, -0.79]* |
| Mean, 9 public sets | 4.936 | 4.757 | 4.708 | 4.661 | -0.05 [-0.09, 0.00] | -0.10 [-0.15, -0.04]* | -0.28 [-0.34, -0.20]* |
| Mean, 7 current leaderboard public sets | 5.747 | 5.532 | 5.484 | 5.422 | -0.06 [-0.12, 0.00]* | -0.11 [-0.18, -0.04]* | -0.33 [-0.41, -0.22]* |
| URGENT2024-Clean (extra, not in means) | 4.29 | 4.21 | 4.15 | 4.19 | +0.04 [-0.07, +0.14] | -0.02 [-0.15, +0.09] | -0.10 [-0.26, +0.06] |
- Where v2 wins: meetings (AMI −0.64 vs base), parliamentary speech (VoxPopuli −1.01), noisy speech (URGENT2024 −0.76) and LibriSpeech test-other (−0.15).
- Where it does not: SPGISpeech is +0.18 vs base (inherited from v1: RL teaches verbatim transcription, and SPGISpeech's references are edited). Earnings22 and Monsoon are unchanged within noise.
- Decoding vs weights: v2's −0.10 over v1 splits about evenly: decoding −0.049*, weights −0.047 (significant on the 7-set mean, −0.062*; on the 9-set mean the CI just touches zero).
Audio it has never seen
To check that the gain is not leaderboard tuning, we evaluated on 11,080 clips (24.8 h) across five evaluation sets from four corpora that were not used in any training, development or selection step: VOiCES (far-field + noise), EasyCom (conversations recorded on AR glasses, with the same utterances on close-talk mics), SLUE-VoxCeleb (web interviews) and OpenSLR 70 (Nigerian English). The evaluation set was locked and hash-checked against all training and leaderboard audio before any model was decoded. Both models use the same beam-3 decoding, so this isolates the weights.
| Set | Clips | v1 weights | v2 | Δ [95% CI] |
|---|---|---|---|---|
| EasyCom, AR-glasses mic (far-field) | 1,535 | 28.25 | 27.47 | −0.78 [−1.29, −0.26]* |
| EasyCom, close-talk mic | 1,535 | 5.78 | 5.59 | −0.19 [−0.42, +0.02] |
| VOiCES (far-field) | 3,000 | 6.66 | 6.66 | 0.00 [−0.08, +0.08] |
| SLUE-VoxCeleb | 3,000 | 5.91 | 5.91 | +0.00 [−0.06, +0.07] |
| Nigerian English (SLR70) | 2,010 | 5.57 | 5.70 | +0.13 [+0.00, +0.26] |
| Mean of 5 sets | 11,080 | 10.44 | 10.27 | −0.17 [−0.29, −0.05]* |
Loop-capped WER (each utterance's errors capped at its reference length), paired bootstrap with 10,000 resamples. v2's gain on fresh audio comes mostly from fewer dropped words (deletions −0.46* per 100 words), concentrated in far-field conversation.
Multilingual check
RL used English audio only. On the first 200 utterances of each of the leaderboard's 16 official multilingual configs (FLEURS / MCV / MLS × de, fr, es, it, pt, nl), v2 matches v1: mean WER 6.14 vs 6.15 (Δ −0.01 [−0.11, +0.13]). This was measured with greedy decoding and without the CJK ban, which must be removed for CJK languages (see Limitations). On 11 non-European FLEURS languages v2 is also level with v1 (9.85 vs 9.89 on FLEURS validation). Like v1, it is somewhat behind the base model on some of those languages, so use the base model for non-English work.
Speed
Beam search costs speed: the 9-set aggregate RTFx is 125 on an A100-40GB, vs 404 for v1 greedy on the same hardware (about 3.2× slower). Set num_beams=1 for greedy speed; v2's weights with greedy decoding have not been measured on the harness.
Quickstart
Transformers
pip install "transformers==5.17.0" torch accelerate soundfile librosa
from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "oruk/orukqwen-asr-1.7b-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
inputs = processor.apply_transcription_request(
audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
language="en",
).to(model.device, model.dtype)
# generation_config.json supplies the v2 decoding (3 beams, no-repeat-8-gram, CJK ban).
output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.decode(generated_ids, return_format="transcription_only")[0])
- Greedy (faster):
model.generate(**inputs, num_beams=1). - Non-English with CJK output: pass
bad_words_ids=Nonetogenerate.
vLLM
The weights use the same model format as v1 (vllm[audio]==0.31.0, LLM(model="oruk/orukqwen-asr-1.7b-v2", max_model_len=2048, limit_mm_per_prompt={"audio": 1})). vLLM's standard SamplingParams path is greedy and ignores the beam settings in generation_config.json, so this standard path uses greedy decoding rather than the reported beam configuration. The results above use the transformers beam-search path.
Timestamps and long audio
Word timestamps use the separate Qwen/Qwen3-ForcedAligner-0.6B-hf model, rather than the ASR weights alone. The processor, tokenizer, architecture and prompt format are unchanged from v1. The reported v2 evaluation is short-form. Do not infer long-form deployment validation of the beam-search configuration from v1's deployment tests; split longer recordings at pauses and validate on the intended workload.
Method
v2 is three stages on top of Qwen/Qwen3-ASR-1.7B.
Stages 1–2 (= v1, unchanged)
- RL: GSPO-token [2] with Levenshtein token-level credit assignment, 64 rollouts per clip and reward
1 − min(WER, 1)on normalized text. - Blend: a uniform soup of three sibling runs [3], blended 50/50 with the base model in the WiSE-FT style [4].
- Details: see the v1 card.
Stage 3: style-pinned GSPO (new in v2)
- Initialisation: v1's weights. Training continues from v1's unrounded fp32 blend.
- Reward: content accuracy minus a style-pin penalty.
- Content accuracy is scored against the human reference after a style-neutral filter: repeats collapsed, fillers and partial words ignored, acronyms joined.
- The style-pin penalty is
λ · m / n_ref_units. Heremcounts style units (fillers, disfluent repeats, letter-spaced acronyms) where the rollout differs from v1's own transcript of the same clip. - Effect: the model is rewarded for getting the words right and penalised for changing how it transcribes in either direction.
- Credit: tokens that carry only style get the group-normalised advantage of the pin term alone. Content tokens get the advantage of the full reward.
- Runs: eight independent runs, each with its own data slice and seed:
- five at λ = 1.0 and three at λ = 2.0;
- each 100 updates at lr 3e-7, from 2,400 transcribed English clips;
- of those clips, 1,600 come from v1's training pool and 800 from a tier whose references were cross-checked against teacher transcripts;
- no reference-model anchor;
- we keep each run's EMA (0.97) weights at update 100.
- Soup: the uniform mean of all eight runs, computed in fp32 and stored in bf16. No member of this chosen eight-run pool was dropped or given extra weight. The decision to pool the λ = 1.0 and λ = 2.0 runs was made after seeing each group's dev results; this is disclosed under Evaluation integrity.
- Trained parameters: as in v1, all decoder blocks, the last two audio-encoder layers and the projector. Embeddings,
lm_headand the rest of the encoder are frozen.
Decoding (new in v2)
| Setting | Value |
|---|---|
num_beams |
3 |
no_repeat_ngram_size |
8 |
length_penalty / repetition_penalty |
1.0 / 1.0 |
bad_words_ids |
32,253 CJK tokens: tokens whose text is CJK, plus byte-level tokens carrying UTF-8 lead bytes 0xE3–0xED |
max_new_tokens |
512 |
These settings were chosen on development data only. The decoding change alone gives −0.049* on the harness (v1 weights + v2 decoding vs v1 greedy).
Training-data disclosure
The upstream Qwen model's pretraining is described by its own model card. The tables here cover Oruk's post-training lineage. All listed audio and references were used for RL prompting/reward computation; no training audio, transcripts or derived datasets are distributed in this repository. References are not used as a token-by-token SFT target. Transcripts from v1 also supply stage 3's style anchor.
Inherited v1 lineage
The inherited lineage drew from a 20,000-clip / 54.6-hour English prompt pool. The archived disclosure counts every pool clip as a conservative usage upper bound, including clips that a training pass might skip. The first 400-update run used 11,304 distinct pool clips; subsequent continuations wrapped through the same ordered pool. Per-source rounded hours may not sum exactly to the rounded total.
| Source | Upstream / catalog | License / terms as recorded | Pool clips / usage upper bound | Pool hours / usage upper bound |
|---|---|---|---|---|
| People's Speech (clean, CC-BY-SA subset) | MLCommons/peoples_speech | Per-item CC-BY-SA-3.0/4.0 clean subset | 9,334 | 36.36 |
| Switchboard-1 Release 2 + transcripts | LDC97S62 + LDC2009T26 (LDC) | LDC97S62+LDC2009T26 | 3,720 | 6.73 |
| Helsinki Prosody (LibriTTS lineage) | mythicinfinity/libritts | CC-BY-4.0 | 1,873 | 3.44 |
| AMI Meeting Corpus (IHM) | edinburghcstr/ami | CC-BY-4.0 | 1,251 | 1.72 |
| SLUE-HVB (Harper Valley Bank) | asapp/slue-phase-2 | Parent Harper Valley terms plus SLUE annotation terms; verify | 749 | 0.80 |
| SLURP (real recordings) | qmeeus/slurp | SLURP audio CC-BY-NC-4.0; text CC-BY-4.0; Hub card missing license | 741 | 0.75 |
| LibriSpeech train-clean-100 | openslr/librispeech_asr | CC-BY-4.0 | 719 | 2.45 |
| ProsodyEmoji | GiulioZh/ProsodyEmoji | CC-BY-4.0 | 360 | 0.53 |
| English Dialects (Google crowdsourced UK/IE) | ylacombe/english_dialects | CC-BY-SA-4.0 | 337 | 0.62 |
| speechocean762 | mispeech/speechocean762 (OpenSLR 101) | CC-BY-4.0 upstream OpenSLR101; Hub card incorrectly reports Apache-2.0 | 296 | 0.37 |
| Nationwide Speech Project | LDC2007S15 (LDC) | LDC2007S15 | 214 | 0.29 |
| DiPCo (Dinner Party Corpus, mixed channel) | benjamin-paine/dinner-party-corpus | CDLA-Permissive-1.0 mirror; verify canonical DiPCo/Zenodo source terms | 130 | 0.20 |
| CAMEO: CREMA-D | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row reports ODbL upstream CREMA-D | 107 | 0.08 |
| MINDS-14 (English) | PolyAI/minds14 | CC-BY-4.0 | 78 | 0.19 |
| CAMEO: eNTERFACE | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row claims MIT upstream, likeness terms need review | 54 | 0.05 |
| CAMEO: JL Corpus | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row reports CC0 upstream JL Corpus | 12 | 0.01 |
| CAMEO: RAVDESS | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate and upstream RAVDESS | 11 | 0.01 |
| Human Voices: Trust and Intent | OSF 45d8j | CC-BY-4.0 | 10 | 0.01 |
| CAMEO: EMNS | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row reports Apache-2.0 upstream EMNS | 4 | 0.01 |
Added style-pinned stage
Each of the eight selected runs has a recorded 2,400-row prompt selection: 1,600 rows from the v6_final pool plus 800 from the teacher-agreement-filtered v8_final/clean tier. This is 19,200 row selections across runs, not 19,200 unique recordings and not a count of gradient-bearing examples. Only groups with useful reward variation contribute to updates, and prompts can be revisited.
The source-level counts below describe the complete candidate pools from which those run selections were drawn. They are conservative per-source availability bounds, not claimed per-source consumed-training totals. We do not report exact per-source consumed counts/hours: the archived selected-run lists use raw IDs that can collide across source partitions, so an ID-only join would misattribute recordings. The two pools also overlap and the inherited v1 pool can overlap them; do not add these columns into a unique-audio total.
v6_final: 34,063 candidate rows / 80.5 hours.v8_final/clean: 15,203 candidate rows / 35.67 hours. This tier retains reliable references using agreement checks from Granite Speech 5.0 2B, Canary-Qwen 2.5B, Parakeet-TDT 0.6B v2 and ARK-ASR 0.6B; their predictions filter candidates rather than replacing the human reference in the style-pinned content reward.- No SPGISpeech data from any split was used for post-training.
| Source | Upstream / catalog | License / terms as recorded | v6 candidate rows | v6 candidate hours | v8 clean candidate rows | v8 clean candidate hours |
|---|---|---|---|---|---|---|
| Switchboard-1 Release 2 + transcripts | LDC97S62 + LDC2009T26 | LDC97S62+LDC2009T26 | 10,614 | 15.93 | 3,910 | 5.73 |
| People’s Speech (dirty CC-BY partition) | MLCommons/peoples_speech | CC-BY (per item, dirty partition) | 7,128 | 28.01 | 2,449 | 9.28 |
| SLURP (real recordings) | qmeeus/slurp | SLURP audio CC-BY-NC-4.0; text CC-BY-4.0; Hub card missing license | 5,462 | 5.52 | 1,876 | 1.80 |
| People’s Speech (clean CC-BY partition) | MLCommons/peoples_speech | CC-BY (per item, clean partition) | 4,923 | 19.08 | 3,037 | 11.29 |
| AMI Meeting Corpus (IHM) | edinburghcstr/ami | CC-BY-4.0 | 3,319 | 4.44 | 2,400 | 3.13 |
| People’s Speech (clean CC-BY-SA subset) | MLCommons/peoples_speech | Per-item CC-BY-SA-3.0/4.0 clean subset | 1,574 | 5.93 | 1,039 | 3.70 |
| English Dialects (Google crowdsourced UK/IE) | ylacombe/english_dialects | CC-BY-SA-4.0 | 387 | 0.69 | 247 | 0.43 |
| speechocean762 | mispeech/speechocean762 / OpenSLR 101 | CC-BY-4.0 upstream OpenSLR101; Hub card incorrectly reports Apache-2.0 | 203 | 0.27 | 68 | 0.09 |
| SLUE-HVB (Harper Valley Bank) | asapp/slue-phase-2 | Parent Harper Valley terms plus SLUE annotation terms; verify | 197 | 0.20 | 89 | 0.09 |
| DiPCo (mixed channel) | benjamin-paine/dinner-party-corpus | CDLA-Permissive-1.0 mirror; verify canonical DiPCo/Zenodo source terms | 165 | 0.25 | 42 | 0.06 |
| Nationwide Speech Project | LDC2007S15 | LDC2007S15 | 54 | 0.07 | 27 | 0.04 |
| MINDS-14 (English) | PolyAI/minds14 | CC-BY-4.0 | 37 | 0.08 | 19 | 0.04 |
Upstream data notices remain as recorded, including LDC-specific terms, non-commercial/share-alike subsets and unresolved mirror/card annotations. Apache-2.0 is the license applied to this model release; it does not relicense those datasets or redistribute them. The source tables retain these terms rather than treating all input data as Apache-2.0.
Evaluation integrity
- Selection on dev data only: every training and soup decision used development data only, never a leaderboard test set. No SPGISpeech data of any split was used for training.
- Dev-to-harness calibration: a dev-to-harness proxy, calibrated on earlier harness runs, predicted v2's weights-only gain as −0.044 ± 0.024. The measured gain was −0.047 (9 sets) and −0.062 (7 sets).
- Measured once: v2 was run once on the harness.
- Disclosed exceptions: v2 missed one pre-registered style guard on dev. Its filler rate was 1.32 per 100 words, vs v1's band floor of 1.37. The leaderboard normaliser removes "uh / um / hmm", so this has no WER cost there.
- SPGISpeech prediction miss: the predicted SPGISpeech change vs v1 + v2 decoding was +0.003. The measured change was +0.025*.
- Fresh-data evaluation: the analysis plan was locked before decoding. The v2 vs v1-weights comparison above is a secondary analysis of that locked set. VOiCES re-records LibriSpeech speakers, some of whose other utterances are in the training pools; both models share this exposure.
Compute and reproducibility
Each continuation trained on one A100 40GB; several independent runs were executed in parallel. We do not report a complete v2 GPU-hour total because that total is not reconciled in the release record. The older v1 lineage used 14.3 A100-hours of training, excluding a shared data-selection pilot of at most 7.5 A100-hours; these figures do not include v2's eight continuations or the wider research campaign.
The reported v2 harness measurements are from an A100-SXM4-40GB. An H200 reproduction and maintainer private-set evaluation are not established by these results. No H200 throughput, production readiness or official board position is claimed. Full WER tables include regressions as well as improvements.
Intended use and limitations
Intended use: English speech-to-text for short-form audio up to ~30 s per request, such as meetings, calls, and read or conversational speech.
Not intended for: speaker identification, or inferring sensitive attributes from voice.
Limitations:
- English first. Other languages work at v1's level but were not optimised. The default CJK ban makes Chinese, Japanese and Korean output impossible: pass
bad_words_ids=None, or use the base model, for those languages. - Verbatim style. v2 transcribes repetitions and false starts. This helps on verbatim references (AMI, VoxPopuli) and costs on edited financial transcripts (SPGISpeech +0.18 vs base).
- Non-speech audio. With the language forced to English, v2 (like v1 and the base model) usually outputs a few words on pure noise or silence. Use voice-activity detection, or auto-language mode, which answers
language Noneon non-speech. - Speed. Beam search is about 3× slower than greedy.
- Formatting. Casing and punctuation follow the base model; the reward is computed on normalised text.
License
Apache-2.0, inherited from the base model. No training data is redistributed with this model.
Citation
@misc{orukqwen2026v2,
title = {orukqwen-asr-1.7b v2: style-pinned GSPO and model soups for speech recognition},
author = {Roll, Nathan and Grenez, Vianney and Marsan, Busra and Yi, Irene},
year = {2026},
note = {oruk research artifact}
}
References
- Qwen Team. Qwen3-ASR Technical Report. arXiv:2601.21337, 2026.
- Zheng et al. Group Sequence Policy Optimization. arXiv:2507.18071, 2025.
- Wortsman et al. Model soups. ICML 2022, arXiv:2203.05482.
- Wortsman et al. Robust fine-tuning of zero-shot models (WiSE-FT). CVPR 2022, arXiv:2109.01903.
Acknowledgements
Built on Qwen3-ASR by the Qwen team. Evaluation uses the Hugging Face Open ASR Leaderboard harness and datasets. The fresh-data evaluation uses VOiCES (Lab41 / SRI), EasyCom (Meta Reality Labs), SLUE-VoxCeleb and OpenSLR 70.