A research artifact from oruk. orukqwen-asr-1.7b is Qwen3-ASR-1.7B [1] post-trained
with our method: reinforcement learning with GSPO [2] and Levenshtein token-level credit assignment, followed by weight-space
blending, i.e. a uniform soup of three sibling RL runs [3] interpolated 50/50 with the base model, in the style of WiSE-FT [4]. The RL
training ran on a single NVIDIA A100 40GB GPU at a time, about 14 GPU-hours in total (plus a one-time data-selection pass of at most 7.5 GPU-hours), drawing from a 54.6-hour pool of transcribed English audio.
Architecture, tokenizer, prompt format and inference code are the same as Qwen/Qwen3-ASR-1.7B-hf; only the weights differ.
Authors: Nathan Roll, Vianney Grenez, Busra Marsan, Irene Yi (oruk)
-hf layout. License: Apache-2.0 (inherited from the base model).r = 1 − min(WER, 1) against the reference, computed on normalized text. A per-word normalizer drops
fillers and non-speech tags. There is no learned judge.A_i = (r_i − mean_{j≠i} r_j) / std(r).
This is the group-normalized estimator used in NVIDIA's Nemotron GSPO recipe [5]. Groups in which all 64 rewards are identical are discarded,
and each update uses 20 informative groups (1,280 transcripts).1/|y|), and there is no KL or SFT term.W_soup is the uniform average of the final weights of E2, E6 and E7 (model soup [3]).W = W_base + 0.5 · (W_soup − W_base), computed in fp32 and stored as bf16. This is a 50/50
interpolation toward the base model in the WiSE-FT style [4]. Frozen tensors are identical to the base model's.Every number below comes from the run logs: the Weights & Biases runtimes, which include the periodic evaluations inside each run, and the data-build logs.
| stage | hardware | time |
|---|---|---|
| Data selection: a one-time 16-sample pilot of the base model over the candidate pool, used to pick the training clips | 3× A100 40GB in parallel, ~2.5 h wall clock | ≤ 7.5 A100-hours (shared by all runs) |
| RL run 1, 400 updates | 1× A100 40GB | 6.67 h |
| Sibling runs E2 / E6 / E7, 150 updates each | 1× A100 40GB each | 3.95 h / 1.80 h / 1.84 h (+0.55 h for a crashed first attempt of E7) |
| Soup + blend | CPU | minutes |
| Lineage total (training) | one A100 40GB at a time | 14.3 A100-hours (14.8 with the crashed attempt), over ~19 h of wall clock |
4.76% mean WER and 718.84 RTFx over the nine public scoring columns. The full English short-form public suite was run on HF Jobs, including both LibriSpeech splits and the additional URGENT2024-Clean set. URGENT2024-Clean is excluded from the nine-set mean. These are contributor-run public results; maintainer private-set evaluation is pending.
Raw predictions, scores, checksums, runtime environments, provenance and launcher. Each run is linked below.
| Dataset | Segments | WER (%) | RTFx | HF Job |
|---|---|---|---|---|
| AMI-Cleaned | 7,715 | 7.92 | 650.77 | run |
| Earnings22-Cleaned-AA-chunked | 341 | 5.88 | 436.47 | run |
| Gigaspeech-Cleaned | 18,757 | 7.12 | 706.04 | run |
| LibriSpeech test-clean | 2,620 | 1.20 | 599.03 | run |
| LibriSpeech test-other | 2,939 | 2.92 | 561.57 | run |
| SPGISpeech | 39,341 | 2.74 | 792.07 | run |
| URGENT2024 | 1,000 | 9.44 | 637.74 | run |
| Voice Arena Monsoon | 2,102 | 3.51 | 532.10 | run |
| Voxpopuli-AA-Cleaned | 628 | 2.07 | 382.79 | run |
| URGENT2024-Clean (extra) | 1,000 | 4.20 | 649.53 | run |
| Nine-set mean / aggregate speed | 75,443 | 4.76 | 718.84 |
Model revision 5925458e58e7b0ac81b9fc36e56d91800d21d987; harness 74497c5ebf1d01b5d39982a12f5b2c399da90e3a (upstream a768a2c7 plus this PR). Official transformers Space, batch 256 on every set, bf16, sdpa, greedy decoding, max_new_tokens=512, the same language="en" prompt, default warmup_steps=10 (up to ten untimed batches), no compilation. Package versions and dataset revision details are in the artifacts. The standard harness filters apply; Earnings22 is scored at parent-session level.
The normalizer/runtime differ from the original A100 run below. No new paired base-model H200 run is claimed.
Official harness (huggingface/open_asr_leaderboard @ edd684247365, transformers/run_eval.py with the flags of
transformers/submit_jobs_qwen3asr.sh, repo normalizer and score_results), greedy decoding, language="en", bf16. WER %.
Our base-model run reproduces the published leaderboard numbers to within 0.02 WER points. The Δ column is a paired per-utterance
bootstrap (2,000 resamples) vs base.
| set | base (Qwen3-ASR-1.7B-hf) | this model | Δ [95% CI] |
|---|---|---|---|
| AMI-Cleaned | 8.32 | 7.92 | -0.40 [-0.50, -0.30]* |
| Earnings22-Cleaned-AA-chunked | 5.84 | 5.85 | +0.01 [-0.40, +0.64] |
| Gigaspeech-Cleaned | 7.22 | 7.11 | -0.11 [-0.14, -0.08]* |
| LS Clean | 1.25 | 1.20 | -0.05 [-0.09, -0.01]* |
| LS Other | 2.94 | 2.89 | -0.05 [-0.12, +0.01] |
| SPGISpeech | 2.58 | 2.75 | +0.17 [+0.15, +0.19]* |
| URGENT2024 | 9.93 | 9.48 | -0.45 [-0.72, -0.20]* |
| Voice Arena Monsoon | 3.48 | 3.50 | +0.02 [-0.06, +0.10] |
| Voxpopuli-AA-Cleaned | 2.86 | 2.11 | -0.75 [-0.95, -0.56]* |
| Average (9 public sets) | 4.94 | 4.76 | -0.18 [-0.24, -0.10]* |
| URGENT2024-Clean (extra, not in average) | 4.29 | 4.21 | |
| RTFx (9 public sets; higher is faster) | 409 | 404 | -1.2% |
* the 95% interval excludes zero.
RTFx was measured on 1x NVIDIA A100-SXM4-40GB (driver 580.178.04; base and this model ran on two VMs of the same type, batch 256 with 128 fallback on OOM). The leaderboard measures RTFx on an NVIDIA H200, so absolute RTFx is not comparable.
This is a held-out English test set from the same sources as training, disjoint by audio hash and recording/speaker group, plus LibriSpeech test-clean and test-other. WER % with the per-word normalizer; paired bootstrap 95% CIs vs base.
| test set | utterances | base | ours | Δ [95% CI] |
|---|---|---|---|---|
| English pooled (18 sets) | 3,004 | 10.09 | 9.09 | -1.00 [-1.20, -0.81]* |
| English macro (mean of 18 sets) | 3,004 | 10.45 | 9.48 | -0.97 [-1.35, -0.67]* |
| LibriSpeech test-clean | 2,620 | 2.19 | 1.89 | -0.30 [-0.39, -0.22]* |
| LibriSpeech test-other | 2,939 | 3.90 | 3.56 | -0.34 [-0.44, -0.24]* |
| AMI Meeting Corpus (IHM) | 200 | 12.80 | 9.21 | -3.59 [-5.71, -1.94]* |
| Human Voices: Trust and Intent | 13 | 8.33 | 5.56 | -2.78 [-8.70, +0.00] |
| SLUE-HVB (Harper Valley Bank) | 200 | 5.89 | 3.67 | -2.22 [-3.27, -1.32]* |
| ProsodyEmoji | 119 | 8.14 | 6.12 | -2.01 [-4.55, -0.34]* |
| People's Speech (clean, CC-BY-SA subset) | 200 | 8.75 | 7.22 | -1.54 [-1.98, -1.13]* |
| DiPCo (Dinner Party Corpus, mixed channel) | 184 | 30.82 | 29.61 | -1.20 [-2.27, -0.32]* |
| Nationwide Speech Project | 200 | 1.31 | 0.23 | -1.08 [-1.84, -0.46]* |
| SLURP (real recordings) | 200 | 12.18 | 11.23 | -0.95 [-1.92, +0.00] |
| CAMEO: JL Corpus | 200 | 7.75 | 6.88 | -0.87 [-1.50, -0.34]* |
| Switchboard-1 Release 2 + transcripts | 200 | 10.23 | 9.39 | -0.84 [-1.31, -0.39]* |
| LibriSpeech train-clean-100 | 200 | 4.75 | 4.30 | -0.45 [-0.79, -0.13]* |
| Helsinki Prosody (LibriTTS lineage) | 200 | 2.24 | 1.98 | -0.26 [-0.65, +0.10] |
| CAMEO: CREMA-D | 200 | 0.42 | 0.25 | -0.17 [-0.52, +0.00] |
| CAMEO: eNTERFACE | 130 | 27.48 | 27.37 | -0.11 [-0.34, +0.00] |
| English Dialects (Google crowdsourced UK/IE) | 200 | 4.08 | 4.00 | -0.07 [-0.37, +0.22] |
| speechocean762 | 200 | 11.36 | 11.36 | +0.00 [-0.59, +0.61] |
| CAMEO: RAVDESS | 60 | 0.83 | 1.11 | +0.28 [+0.00, +0.83] |
| MINDS-14 (English) | 98 | 30.74 | 31.06 | +0.32 [-0.07, +0.85] |
RL used English audio only, so we checked that the other languages did not regress. We ran the leaderboard's multilingual script (transformers/run_eval_ml.py, language given explicitly, max_new_tokens=256, multilingual scorer) on a small sample: the first 200 test utterances of 5 FLEURS and 2 MLS sets. WER %, paired bootstrap 95% CI vs base.
| set (first 200 test utterances) | base | this model | Δ [95% CI] |
|---|---|---|---|
| FLEURS German | 2.82 | 2.76 | -0.07 [-0.37, +0.21] |
| FLEURS French | 4.25 | 4.23 | -0.02 [-0.40, +0.37] |
| FLEURS Spanish | 2.30 | 2.28 | -0.02 [-0.21, +0.17] |
| FLEURS Italian | 1.46 | 1.53 | +0.06 [-0.08, +0.21] |
| FLEURS Portuguese | 4.05 | 3.88 | -0.16 [-0.47, +0.12] |
| MLS Spanish | 3.49 | 3.43 | -0.06 [-0.27, +0.14] |
| MLS French | 5.62 | 5.46 | -0.16 [-0.47, +0.15] |
| mean of 7 | 3.43 | 3.37 | -0.06 |
We found no measurable regression. No set changed significantly, and no output was empty or in the wrong script. Language auto-detection (no language hint) still returns English and Chinese correctly on the base model's demo clips. This is a small-sample check, not a full multilingual evaluation.
pip install "transformers==5.17.0" torch accelerate soundfile librosa
from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "oruk/orukqwen-asr-1.7b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
inputs = processor.apply_transcription_request(
audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
language="en",
).to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.decode(generated_ids, return_format="transcription_only")[0])
Batch inference (pass a list of audio paths/URLs/arrays; same prompt for every item):
from transformers import AutoProcessor, AutoModelForMultimodalLM
model_id = "oruk/orukqwen-asr-1.7b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")
audio = [
"https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
"https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
]
inputs = processor.apply_transcription_request(audio, language="en").to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
for text in processor.decode(generated_ids, return_format="transcription_only"):
print(text)
pip install "vllm[audio]==0.31.0" librosa
import librosa
from huggingface_hub import hf_hub_download
from vllm import LLM, SamplingParams
llm = LLM(model="oruk/orukqwen-asr-1.7b", max_model_len=2048, limit_mm_per_prompt={"audio": 1})
prompt = "<|im_start|>user\n<|audio_start|><|audio_pad|><|audio_end|><|im_end|>\n<|im_start|>assistant\nlanguage English<asr_text>"
path = hf_hub_download("bezzam/audio_samples", "librispeech_mr_quilter.wav", repo_type="dataset")
audio, sr = librosa.load(path, sr=16000)
out = llm.generate({"prompt": prompt, "multi_modal_data": {"audio": (audio, sr)}},
SamplingParams(temperature=0.0, max_tokens=256))
print(out[0].outputs[0].text)
vllm serve oruk/orukqwen-asr-1.7b --max-model-len 2048 --port 8000
curl -s -L -o sample.wav https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav
curl -s http://localhost:8000/v1/audio/transcriptions \
-F file=@sample.wav -F model=oruk/orukqwen-asr-1.7b -F language=en
The model is trained and evaluated on clips of up to 30 s. Longer recordings can still be passed in one call. We tested 70 s, 136 s and 322 s recordings (concatenated LibriSpeech test-clean). With transformers, apply_transcription_request handles them as-is (WER 1.2-2.2%, the same as the base model). With vLLM they work once max_model_len is raised (we used 8192 for 5 minutes; the 2048 in the snippets above is enough for short-form audio). For hour-long audio, split it into segments of up to a few minutes, preferably at pauses.
Word-level timestamps come from the separate forced aligner, exactly as for the base model
(Qwen/Qwen3-ForcedAligner-0.6B-hf). The aligner is unchanged and works on this model's transcripts exactly as on the base model's. The snippet below is run verbatim in our deployment tests.
import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM, AutoModelForTokenClassification
asr_model_id = "oruk/orukqwen-asr-1.7b"
aligner_model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"
asr_processor = AutoProcessor.from_pretrained(asr_model_id)
asr_model = AutoModelForMultimodalLM.from_pretrained(asr_model_id, device_map="auto")
aligner_processor = AutoProcessor.from_pretrained(aligner_model_id)
aligner_model = AutoModelForTokenClassification.from_pretrained(aligner_model_id, dtype=torch.bfloat16, device_map="auto")
audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = asr_processor.apply_transcription_request(audio=audio_url, language="en").to(asr_model.device, asr_model.dtype)
output_ids = asr_model.generate(**inputs, max_new_tokens=256)
transcript = asr_processor.decode(output_ids[:, inputs["input_ids"].shape[1]:], return_format="transcription_only")[0]
aligner_inputs, word_lists = aligner_processor.prepare_forced_aligner_inputs(
audio=audio_url, transcript=transcript, language="English",
)
aligner_inputs = aligner_inputs.to(aligner_model.device, aligner_model.dtype)
with torch.inference_mode():
outputs = aligner_model(**aligner_inputs)
timestamps = aligner_processor.decode_forced_alignment(
logits=outputs.logits, input_ids=aligner_inputs["input_ids"], word_lists=word_lists,
timestamp_token_id=aligner_model.config.timestamp_token_id,
)[0]
for item in timestamps:
print(f"{item['text']:<20} {item['start_time']:>8.3f}s -> {item['end_time']:>8.3f}s")
English speech-to-text for short-form audio (up to ~30 s per request; see Long audio): meetings, calls, read and conversational speech. The base model is multilingual, while the RL post-training used English audio only (see Limitations). This model is released as a research artifact for studying RL post-training and weight-space blending for ASR. It is not intended for speaker identification or for inferring sensitive attributes from voice.
Apache-2.0, inherited from the base model Qwen/Qwen3-ASR-1.7B. No training data is redistributed with this model.
@misc{orukqwen2026,
title = {orukqwen-asr-1.7b: GSPO with Levenshtein token-level credit assignment and weight-space blending for speech recognition},
author = {Roll, Nathan and Grenez, Vianney and Marsan, Busra and Yi, Irene},
year = {2026},
howpublished = {\url{https://huggingface.co/oruk/orukqwen-asr-1.7b}},
note = {oruk research artifact}
}
Built on Qwen3-ASR by the Qwen team. The training audio comes from: People's Speech / MLCommons, LibriSpeech / LibriTTS, the LDC (Switchboard-1, Nationwide Speech Project), AMI, SLURP, SLUE, DiPCo, speechocean762, MINDS-14, CAMEO and others. Evaluation uses the Hugging Face Open ASR Leaderboard harness and datasets.
Qwen/Qwen3-ASR-1.7B