Eight RL runs averaged into one speech-recognition model. v2 continues our RL post-training of Qwen3-ASR-1.7B [1] with a second, style-pinned GSPO stage, averages eight independent runs into one model, and ships with a beam-search decoding config.
| Result | What it measures |
|---|---|
| 4.66 mean WER | 9 public English sets, official harness (v1: 4.76 · base: 4.94) |
| 5.42 mean WER | equal-weight mean over 7 public English test sets in our November 7 official-harness run; see the dated comparison below |
| −0.17* WER | vs v1's weights on 11,080 clips from 5 corpora used nowhere in training or tuning (−0.78* on AR-glasses far-field audio) |
| 2.04B parameters | same architecture, tokenizer and prompt as Qwen/Qwen3-ASR-1.7B-hf; the default decoding configuration changes from v1 |
Authors: Nathan Roll, Vianney Grenez, Busra Marsan and Irene Yi (oruk).
The model's 1.7B name follows the Qwen model family; its complete parameter count is 2.04B. It is the same architecture as the base model, rather than an ensemble at inference.
| Model | Mean WER over the same 7 public sets (%) | Source |
|---|---|---|
| OrukQwen 2 | 5.4219 | Our run of the official harness on November 7, 2026 with the shipped 3-beam decoding configuration |
| NVIDIA Nemotron-3.5-Transcribe-30B-A4B | 5.5114 | Equal-weight mean of displayed board columns read November 9, 2026 |
| Qwen3-ASR-1.7B-hf | 5.7457 | Equal-weight mean of displayed board columns read November 9, 2026 |
The seven sets are AMI-Cleaned, Earnings22-Cleaned-AA-chunked, Gigaspeech-Cleaned, SPGISpeech, URGENT2024, Voice Arena Monsoon and Voxpopuli-AA-Cleaned. These are equal-weight means of dataset WERs rather than pooled word counts. Comparator columns are rounded by the board; the extra digits above describe their arithmetic mean rather than extra measurement precision. Our own base-model run is 5.747% on these sets.
This is a dated public-set comparison and not an official leaderboard rank. The board's ranking includes two private test-set aggregates that its maintainers run. We do not claim that the maintainers have evaluated or accepted this release. The board has changed since the comparison date; this card makes no current “#1” claim.
generation_config.json now uses 3-beam search, a no-repeat-8-gram guard against loops, and a ban on CJK tokens, so English output cannot drift into Chinese/Japanese/Korean script.Each part helps on its own, and they stack:
Official harness (huggingface/open_asr_leaderboard @ edd684247365, transformers/run_eval.py unchanged, repo normalizer and scorer), bf16, language="en", 1× A100-SXM4-40GB. Measured once, on 2026-11-07. WER %, lower is better. Δ columns are paired per-utterance bootstraps (2,000 resamples); * = the 95% interval excludes zero. The best value in each row is in bold.
| Set | Base Qwen3-ASR-1.7B | v1 (greedy) | v1 weights + v2 decoding | v2 | Δ v2 vs v1 weights at the same decoding | Δ v2 vs v1 | Δ v2 vs base |
|---|---|---|---|---|---|---|---|
| AMI-Cleaned | 8.32 | 7.92 | 7.96 | 7.68 | -0.28 [-0.37, -0.18]* | -0.24 [-0.32, -0.15]* | -0.64 [-0.75, -0.51]* |
| Earnings22-Cleaned-AA-chunked | 5.84 | 5.85 | 5.74 | 5.89 | +0.16 [-0.07, +0.49] | +0.04 [-0.39, +0.40] | +0.05 [-0.37, +0.65] |
| Gigaspeech-Cleaned | 7.22 | 7.11 | 7.08 | 7.10 | +0.02 [-0.01, +0.04] | -0.01 [-0.04, +0.01] | -0.12 [-0.15, -0.09]* |
| LibriSpeech test-clean | 1.25 | 1.20 | 1.17 | 1.19 | +0.02 [-0.01, +0.06] | 0.00 [-0.04, +0.04] | -0.06 [-0.10, -0.01]* |
| LibriSpeech test-other | 2.94 | 2.89 | 2.81 | 2.80 | -0.01 [-0.07, +0.05] | -0.09 [-0.16, -0.03]* | -0.15 [-0.23, -0.07]* |
| SPGISpeech | 2.58 | 2.75 | 2.73 | 2.75 | +0.03 [+0.01, +0.04]* | +0.01 [0.00, +0.02] | +0.18 [+0.16, +0.20]* |
| URGENT2024 | 9.93 | 9.48 | 9.37 | 9.17 | -0.19 [-0.41, +0.03] | -0.31 [-0.55, -0.05]* | -0.76 [-1.11, -0.42]* |
| Voice Arena Monsoon | 3.48 | 3.50 | 3.51 | 3.50 | -0.01 [-0.08, +0.06] | 0.00 [-0.08, +0.07] | +0.02 [-0.07, +0.11] |
| Voxpopuli-AA-Cleaned | 2.86 | 2.11 | 2.01 | 1.86 | -0.15 [-0.28, -0.03]* | -0.25 [-0.39, -0.12]* | -1.01 [-1.22, -0.79]* |
| Mean, 9 public sets | 4.936 | 4.757 | 4.708 | 4.661 | -0.05 [-0.09, 0.00] | -0.10 [-0.15, -0.04]* | -0.28 [-0.34, -0.20]* |
| Mean, 7 current leaderboard public sets | 5.747 | 5.532 | 5.484 | 5.422 | -0.06 [-0.12, 0.00]* | -0.11 [-0.18, -0.04]* | -0.33 [-0.41, -0.22]* |
| URGENT2024-Clean (extra, not in means) | 4.29 | 4.21 | 4.15 | 4.19 | +0.04 [-0.07, +0.14] | -0.02 [-0.15, +0.09] | -0.10 [-0.26, +0.06] |
To check that the gain is not leaderboard tuning, we evaluated on 11,080 clips (24.8 h) across five evaluation sets from four corpora that were not used in any training, development or selection step: VOiCES (far-field + noise), EasyCom (conversations recorded on AR glasses, with the same utterances on close-talk mics), SLUE-VoxCeleb (web interviews) and OpenSLR 70 (Nigerian English). The evaluation set was locked and hash-checked against all training and leaderboard audio before any model was decoded. Both models use the same beam-3 decoding, so this isolates the weights.
| Set | Clips | v1 weights | v2 | Δ [95% CI] |
|---|---|---|---|---|
| EasyCom, AR-glasses mic (far-field) | 1,535 | 28.25 | 27.47 | −0.78 [−1.29, −0.26]* |
| EasyCom, close-talk mic | 1,535 | 5.78 | 5.59 | −0.19 [−0.42, +0.02] |
| VOiCES (far-field) | 3,000 | 6.66 | 6.66 | 0.00 [−0.08, +0.08] |
| SLUE-VoxCeleb | 3,000 | 5.91 | 5.91 | +0.00 [−0.06, +0.07] |
| Nigerian English (SLR70) | 2,010 | 5.57 | 5.70 | +0.13 [+0.00, +0.26] |
| Mean of 5 sets | 11,080 | 10.44 | 10.27 | −0.17 [−0.29, −0.05]* |
Loop-capped WER (each utterance's errors capped at its reference length), paired bootstrap with 10,000 resamples. v2's gain on fresh audio comes mostly from fewer dropped words (deletions −0.46* per 100 words), concentrated in far-field conversation.
RL used English audio only. On the first 200 utterances of each of the leaderboard's 16 official multilingual configs (FLEURS / MCV / MLS × de, fr, es, it, pt, nl), v2 matches v1: mean WER 6.14 vs 6.15 (Δ −0.01 [−0.11, +0.13]). This was measured with greedy decoding and without the CJK ban, which must be removed for CJK languages (see Limitations). On 11 non-European FLEURS languages v2 is also level with v1 (9.85 vs 9.89 on FLEURS validation). Like v1, it is somewhat behind the base model on some of those languages, so use the base model for non-English work.
Beam search costs speed: the 9-set aggregate RTFx is 125 on an A100-40GB, vs 404 for v1 greedy on the same hardware (about 3.2× slower). Set num_beams=1 for greedy speed; v2's weights with greedy decoding have not been measured on the harness.
pip install "transformers==5.17.0" torch accelerate soundfile librosa
from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "oruk/orukqwen-asr-1.7b-v2"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
inputs = processor.apply_transcription_request(
audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
language="en",
).to(model.device, model.dtype)
# generation_config.json supplies the v2 decoding (3 beams, no-repeat-8-gram, CJK ban).
output_ids = model.generate(**inputs)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.decode(generated_ids, return_format="transcription_only")[0])
model.generate(**inputs, num_beams=1).bad_words_ids=None to generate.The weights use the same model format as v1 (vllm[audio]==0.31.0, LLM(model="oruk/orukqwen-asr-1.7b-v2", max_model_len=2048, limit_mm_per_prompt={"audio": 1})). vLLM's standard SamplingParams path is greedy and ignores the beam settings in generation_config.json, so this standard path uses greedy decoding rather than the reported beam configuration. The results above use the transformers beam-search path.
Word timestamps use the separate Qwen/Qwen3-ForcedAligner-0.6B-hf model, rather than the ASR weights alone. The processor, tokenizer, architecture and prompt format are unchanged from v1. The reported v2 evaluation is short-form. Do not infer long-form deployment validation of the beam-search configuration from v1's deployment tests; split longer recordings at pauses and validate on the intended workload.
v2 is three stages on top of Qwen/Qwen3-ASR-1.7B.
1 − min(WER, 1) on normalized text.λ · m / n_ref_units. Here m counts style units (fillers, disfluent repeats, letter-spaced acronyms) where the rollout differs from v1's own transcript of the same clip.lm_head and the rest of the encoder are frozen.| Setting | Value |
|---|---|
num_beams |
3 |
no_repeat_ngram_size |
8 |
length_penalty / repetition_penalty |
1.0 / 1.0 |
bad_words_ids |
32,253 CJK tokens: tokens whose text is CJK, plus byte-level tokens carrying UTF-8 lead bytes 0xE3–0xED |
max_new_tokens |
512 |
These settings were chosen on development data only. The decoding change alone gives −0.049* on the harness (v1 weights + v2 decoding vs v1 greedy).
The upstream Qwen model's pretraining is described by its own model card. The tables here cover Oruk's post-training lineage. All listed audio and references were used for RL prompting/reward computation; no training audio, transcripts or derived datasets are distributed in this repository. References are not used as a token-by-token SFT target. Transcripts from v1 also supply stage 3's style anchor.
The inherited lineage drew from a 20,000-clip / 54.6-hour English prompt pool. The archived disclosure counts every pool clip as a conservative usage upper bound, including clips that a training pass might skip. The first 400-update run used 11,304 distinct pool clips; subsequent continuations wrapped through the same ordered pool. Per-source rounded hours may not sum exactly to the rounded total.
| Source | Upstream / catalog | License / terms as recorded | Pool clips / usage upper bound | Pool hours / usage upper bound |
|---|---|---|---|---|
| People's Speech (clean, CC-BY-SA subset) | MLCommons/peoples_speech | Per-item CC-BY-SA-3.0/4.0 clean subset | 9,334 | 36.36 |
| Switchboard-1 Release 2 + transcripts | LDC97S62 + LDC2009T26 (LDC) | LDC97S62+LDC2009T26 | 3,720 | 6.73 |
| Helsinki Prosody (LibriTTS lineage) | mythicinfinity/libritts | CC-BY-4.0 | 1,873 | 3.44 |
| AMI Meeting Corpus (IHM) | edinburghcstr/ami | CC-BY-4.0 | 1,251 | 1.72 |
| SLUE-HVB (Harper Valley Bank) | asapp/slue-phase-2 | Parent Harper Valley terms plus SLUE annotation terms; verify | 749 | 0.80 |
| SLURP (real recordings) | qmeeus/slurp | SLURP audio CC-BY-NC-4.0; text CC-BY-4.0; Hub card missing license | 741 | 0.75 |
| LibriSpeech train-clean-100 | openslr/librispeech_asr | CC-BY-4.0 | 719 | 2.45 |
| ProsodyEmoji | GiulioZh/ProsodyEmoji | CC-BY-4.0 | 360 | 0.53 |
| English Dialects (Google crowdsourced UK/IE) | ylacombe/english_dialects | CC-BY-SA-4.0 | 337 | 0.62 |
| speechocean762 | mispeech/speechocean762 (OpenSLR 101) | CC-BY-4.0 upstream OpenSLR101; Hub card incorrectly reports Apache-2.0 | 296 | 0.37 |
| Nationwide Speech Project | LDC2007S15 (LDC) | LDC2007S15 | 214 | 0.29 |
| DiPCo (Dinner Party Corpus, mixed channel) | benjamin-paine/dinner-party-corpus | CDLA-Permissive-1.0 mirror; verify canonical DiPCo/Zenodo source terms | 130 | 0.20 |
| CAMEO: CREMA-D | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row reports ODbL upstream CREMA-D | 107 | 0.08 |
| MINDS-14 (English) | PolyAI/minds14 | CC-BY-4.0 | 78 | 0.19 |
| CAMEO: eNTERFACE | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row claims MIT upstream, likeness terms need review | 54 | 0.05 |
| CAMEO: JL Corpus | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row reports CC0 upstream JL Corpus | 12 | 0.01 |
| CAMEO: RAVDESS | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate and upstream RAVDESS | 11 | 0.01 |
| Human Voices: Trust and Intent | OSF 45d8j | CC-BY-4.0 | 10 | 0.01 |
| CAMEO: EMNS | amu-cai/CAMEO | CC-BY-NC-SA-4.0 aggregate; row reports Apache-2.0 upstream EMNS | 4 | 0.01 |
Each of the eight selected runs has a recorded 2,400-row prompt selection: 1,600 rows from the v6_final pool plus 800 from the teacher-agreement-filtered v8_final/clean tier. This is 19,200 row selections across runs, not 19,200 unique recordings and not a count of gradient-bearing examples. Only groups with useful reward variation contribute to updates, and prompts can be revisited.
The source-level counts below describe the complete candidate pools from which those run selections were drawn. They are conservative per-source availability bounds, not claimed per-source consumed-training totals. We do not report exact per-source consumed counts/hours: the archived selected-run lists use raw IDs that can collide across source partitions, so an ID-only join would misattribute recordings. The two pools also overlap and the inherited v1 pool can overlap them; do not add these columns into a unique-audio total.
v6_final: 34,063 candidate rows / 80.5 hours.v8_final/clean: 15,203 candidate rows / 35.67 hours. This tier retains reliable references using agreement checks from Granite Speech 5.0 2B, Canary-Qwen 2.5B, Parakeet-TDT 0.6B v2 and ARK-ASR 0.6B; their predictions filter candidates rather than replacing the human reference in the style-pinned content reward.| Source | Upstream / catalog | License / terms as recorded | v6 candidate rows | v6 candidate hours | v8 clean candidate rows | v8 clean candidate hours |
|---|---|---|---|---|---|---|
| Switchboard-1 Release 2 + transcripts | LDC97S62 + LDC2009T26 | LDC97S62+LDC2009T26 | 10,614 | 15.93 | 3,910 | 5.73 |
| People’s Speech (dirty CC-BY partition) | MLCommons/peoples_speech | CC-BY (per item, dirty partition) | 7,128 | 28.01 | 2,449 | 9.28 |
| SLURP (real recordings) | qmeeus/slurp | SLURP audio CC-BY-NC-4.0; text CC-BY-4.0; Hub card missing license | 5,462 | 5.52 | 1,876 | 1.80 |
| People’s Speech (clean CC-BY partition) | MLCommons/peoples_speech | CC-BY (per item, clean partition) | 4,923 | 19.08 | 3,037 | 11.29 |
| AMI Meeting Corpus (IHM) | edinburghcstr/ami | CC-BY-4.0 | 3,319 | 4.44 | 2,400 | 3.13 |
| People’s Speech (clean CC-BY-SA subset) | MLCommons/peoples_speech | Per-item CC-BY-SA-3.0/4.0 clean subset | 1,574 | 5.93 | 1,039 | 3.70 |
| English Dialects (Google crowdsourced UK/IE) | ylacombe/english_dialects | CC-BY-SA-4.0 | 387 | 0.69 | 247 | 0.43 |
| speechocean762 | mispeech/speechocean762 / OpenSLR 101 | CC-BY-4.0 upstream OpenSLR101; Hub card incorrectly reports Apache-2.0 | 203 | 0.27 | 68 | 0.09 |
| SLUE-HVB (Harper Valley Bank) | asapp/slue-phase-2 | Parent Harper Valley terms plus SLUE annotation terms; verify | 197 | 0.20 | 89 | 0.09 |
| DiPCo (mixed channel) | benjamin-paine/dinner-party-corpus | CDLA-Permissive-1.0 mirror; verify canonical DiPCo/Zenodo source terms | 165 | 0.25 | 42 | 0.06 |
| Nationwide Speech Project | LDC2007S15 | LDC2007S15 | 54 | 0.07 | 27 | 0.04 |
| MINDS-14 (English) | PolyAI/minds14 | CC-BY-4.0 | 37 | 0.08 | 19 | 0.04 |
Upstream data notices remain as recorded, including LDC-specific terms, non-commercial/share-alike subsets and unresolved mirror/card annotations. Apache-2.0 is the license applied to this model release; it does not relicense those datasets or redistribute them. The source tables retain these terms rather than treating all input data as Apache-2.0.
Each continuation trained on one A100 40GB; several independent runs were executed in parallel. We do not report a complete v2 GPU-hour total because that total is not reconciled in the release record. The older v1 lineage used 14.3 A100-hours of training, excluding a shared data-selection pilot of at most 7.5 A100-hours; these figures do not include v2's eight continuations or the wider research campaign.
The reported v2 harness measurements are from an A100-SXM4-40GB. An H200 reproduction and maintainer private-set evaluation are not established by these results. No H200 throughput, production readiness or official board position is claimed. Full WER tables include regressions as well as improvements.
Intended use: English speech-to-text for short-form audio up to ~30 s per request, such as meetings, calls, and read or conversational speech.
Not intended for: speaker identification, or inferring sensitive attributes from voice.
Limitations:
bad_words_ids=None, or use the base model, for those languages.language None on non-speech.Apache-2.0, inherited from the base model. No training data is redistributed with this model.
@misc{orukqwen2026v2,
title = {orukqwen-asr-1.7b v2: style-pinned GSPO and model soups for speech recognition},
author = {Roll, Nathan and Grenez, Vianney and Marsan, Busra and Yi, Irene},
year = {2026},
note = {oruk research artifact}
}
Built on Qwen3-ASR by the Qwen team. Evaluation uses the Hugging Face Open ASR Leaderboard harness and datasets. The fresh-data evaluation uses VOiCES (Lab41 / SRI), EasyCom (Meta Reality Labs), SLUE-VoxCeleb and OpenSLR 70.
Qwen/Qwen3-ASR-1.7B