Wav2Vec2-base for 35 spoken commands

facebook/wav2vec2-base fine-tuned on Speech Commands v0.02 to recognise 35 spoken English words: yes, no, up, down, left, right, on, off, stop, go, forward, backward, follow, learn, the digits zero to nine, and bed, bird, cat, dog, happy, house, marvin, sheila, tree, visual, wow.

Try it: speech-commands-demo · Code: github.com/dpontejs/classificacao_voz

Usage

from transformers import pipeline

classifier = pipeline("audio-classification", model="diegomponte/wav2vec2-base-speech-commands-35")
classifier("my_recording.wav", top_k=3)
# [{'label': 'yes', 'score': 0.99}, ...]

In the browser, with Transformers.js and the int8 ONNX export in onnx/:

import { pipeline } from "@huggingface/transformers";

const classifier = await pipeline(
  "audio-classification", "diegomponte/wav2vec2-base-speech-commands-35", { dtype: "q8" },
);
await classifier(float32Audio16kHz, { top_k: 3 });

Input should be one second of 16 kHz mono audio with a single word. For longer recordings, cut the second around the word first (the demo keeps the loudest second).

Results

Official test split: 11,005 clips from speakers never seen in training.

The ONNX exports were measured on the same test split with ONNX Runtime:

Quantizing every layer, including the convolutional feature encoder, gave a 95 MB file but dropped accuracy to 78.4%, so the convolutions stay in fp32.

Hardest words by F1: "tree" (0.91), "forward" (0.95), "learn" (0.95). Most errors are between words that sound alike: "tree" ↔ "three" (25 of the errors) and "off" heard as "up" (10).

This is the 35-word task. It is not comparable to the 12-class keyword-spotting benchmark (10 words + "unknown" + "silence") reported by SUPERB and many papers.

Training

• Data: the full training split (84,843 clips) with the official validation and test lists, so no speaker appears in two splits. Clips are zero-padded to one second and normalised by the Wav2Vec2 feature extractor.
• Model: Wav2Vec2ForSequenceClassification (projection to 256, mean pooling over time, linear classifier). The convolutional feature encoder is frozen; the transformer and head are trained.
• Hyperparameters: AdamW, learning rate 3e-5, 10% linear warmup, batch 32, fp16, 5 epochs, seed 42. Best epoch chosen by validation accuracy.
• Hardware: one RTX 4050 laptop GPU (6 GB), 35 minutes.

Limitations

• English only, single isolated words, one second long. It does not transcribe speech.
• It always picks one of the 35 words: there is no "unknown" or "silence" class, so any other sound still gets a label. Use the score as a confidence threshold if that matters.
• Speech Commands was recorded by volunteers through browsers and phones; accents, children's voices and noisy rooms are less represented than in real use.

License and credits

Apache-2.0, like the base model facebook/wav2vec2-base. Trained on Speech Commands by Pete Warden, released under CC-BY-4.0.