SMAD CRNN — speech / music / singing discrimination

A 834k-parameter CRNN that labels a 4-second audio clip as spoken voice over noise, spoken voice over music, sung voice over music, or no voice. Trained from scratch on log-mel spectrograms — no pretrained backbone, no fine-tuning. Runs on CPU.

▶ Try it — upload a file, get a colour-coded timeline

The point of the taxonomy: most speech/music segmenters merge "someone is talking over a music bed" with "someone is singing", or call singing music and stop there. SMAD separates those, which is what you need for lyric/dialogue routing, dubbing QC, or MV vs. interview classification.

Labels

Input: mono, 16 kHz, 4-second windows. Output: 4 logits; divide by config.temperature (0.7095) before softmax for calibrated probabilities.

Quick start

import librosa, torch
from transformers import AutoFeatureExtractor, AutoModelForAudioClassification

model_id = "duclvQ/smad"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForAudioClassification.from_pretrained(model_id, trust_remote_code=True).eval()

audio, _ = librosa.load("clip.mp3", sr=16000, mono=True)   # mono float32 @ 16 kHz

inputs = feature_extractor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
probs = torch.softmax(logits / model.config.temperature, dim=-1)[0]

label_id = int(probs.argmax())
print(model.config.id2label[label_id], float(probs[label_id]))

The feature extractor pads or truncates to exactly 4 seconds. For anything longer, use the helper below.

Audio of any length: sliding window + smoothing

Copy-paste. No dependencies beyond librosa, numpy, torch, transformers.

import librosa, numpy as np, torch


def analyze(path, model, feature_extractor,
            hop_seconds=1.0, smooth_windows=5, batch_size=64):
    """Label a file of any length. Returns merged segments:
    [{"start": 0.0, "end": 12.0, "label": "singing_music", "confidence": 0.93}, ...]

    hop_seconds    step between windows (window itself is fixed at 4 s by training)
    smooth_windows median filter width over time; 1 disables smoothing
    """
    sr = feature_extractor.sampling_rate
    win = int(model.config.segment_seconds * sr)
    hop = max(int(hop_seconds * sr), 1)

    audio, _ = librosa.load(path, sr=sr, mono=True)
    if len(audio) < win:
        audio = np.pad(audio, (0, win - len(audio)))
    starts = list(range(0, len(audio) - win + 1, hop))

    probs = []
    for i in range(0, len(starts), batch_size):
        batch = [audio[s:s + win] for s in starts[i:i + batch_size]]
        inputs = feature_extractor(batch, sampling_rate=sr, return_tensors="pt")
        with torch.no_grad():
            logits = model(**inputs).logits
        probs.append(torch.softmax(logits / model.config.temperature, dim=-1))
    probs = torch.cat(probs).numpy()

    # Median filter across time: kills isolated one-window flips, which are the
    # dominant error mode on real audio (a single drum fill read as `none`).
    if smooth_windows > 1:
        pad = smooth_windows // 2
        padded = np.pad(probs, ((pad, pad), (0, 0)), mode="edge")
        probs = np.stack([np.median(padded[i:i + smooth_windows], axis=0)
                          for i in range(len(probs))])

    # Merge runs of equal labels. A run ends where the next one starts, so the
    # returned segments tile the audio without overlapping -- consecutive
    # windows overlap by design whenever hop_seconds < 4.
    segments = []
    for start, p in zip(starts, probs):
        cid = int(p.argmax())
        if segments and segments[-1]["_id"] == cid:
            segments[-1]["_conf"].append(float(p[cid]))
        else:
            segments.append({"start": start / sr, "_id": cid, "_conf": [float(p[cid])]})

    duration = len(audio) / sr
    for cur, nxt in zip(segments, segments[1:]):
        cur["end"] = nxt["start"]
    segments[-1]["end"] = min(starts[-1] / sr + win / sr, duration)

    return [{"start": round(s["start"], 2), "end": round(s["end"], 2),
             "label": model.config.id2label[s["_id"]],
             "confidence": round(float(np.mean(s["_conf"])), 4)}
            for s in segments]


for seg in analyze("song.mp3", model, feature_extractor):
    print(f"{seg['start']:7.1f} - {seg['end']:7.1f}s  {seg['label']:<14s} {seg['confidence']:.2f}")

Segment boundaries are quantised to hop_seconds, and a boundary is only ever accurate to within one 4-second window — that window is the model's unit of decision, not a frame-accurate onset. Lower hop_seconds for finer boundaries at linear cost; raise smooth_windows if the timeline flickers.

Results

4,000 held-out clips, 1,000 per class, source-disjoint from training (no speaker, song, or noise file appears in both). Chance is 25%.

Overall accuracy: 87.95% · macro F1 0.879 · 834,728 parameters

Per class

Confusion matrix

Rows = true, columns = predicted.

The two speech classes are close to solved. singing_music ↔ none holds 372 of the 482 total errors — 77%. Telling a sung voice apart from the instrumental track under it is the open problem, not an implementation gap. If your use case only needs "is anyone talking", the relevant number is the top-left 2×2 block.

Accuracy by mixing difficulty

The test mixer records the SNR (or gain, for single-source clips) of each clip:

Voice buried at negative SNR is not the hard case. Single-source clips are — they are the none class, where there is no second source to key on.

Calibration

Trained with label smoothing 0.1 + SpecAugment, then temperature-scaled on the validation split (T = 0.7095, fitted — not tuned by hand).

Why this matters: an earlier transformer trained without label smoothing put

0.99 confidence on 24.5% of its own mistakes — its confidence score was useless as a filter, which is the first thing anyone actually wants from it. This model's score is usable as a threshold. Temperature scaling is monotonic, so it never changes which class wins.

Compared with inaSpeechSegmenter

Same 1,000 clips, drawn from the test split above. The two systems don't share a taxonomy — inaSpeechSegmenter tags singing as music by design, and merges speech-over-music with speech-over-noise — so a naive head-to-head would be meaningless. Two separate measurements instead:

Task A — is a spoken voice present? The question both systems are built to answer. Neither is handicapped.

Task B — is any voice present, spoken or sung? SMAD is built for this; inaSpeechSegmenter is built not to distinguish it. Reporting B is not a criticism of their tool — it quantifies what the extra class buys, which is the only reason to carry it.

On the 250 clips that genuinely contain singing, inaSpeechSegmenter labelled 228 music, 14 noise, 7 noEnergy, and 1 speech. SMAD got 188 right.

Size is comparable, not smaller: SMAD is 834,728 parameters (3.3 MB) against inaSpeechSegmenter's speech/music/noise CNN at ~789k stored values (3.2 MB). The gain here is the taxonomy and the recall, not the footprint.

Caveat, stated plainly: the clips come from SMAD's own synthetic mixer, so this is home turf. inaSpeechSegmenter was trained on different data for a different label set and never saw this distribution. Read this as "what the extra class buys on this task", not as a general ranking.

Training data

40,000 synthetic 4-second mixtures. The classes are defined by what is mixed together, so segments are synthesised from clean stems rather than scraped from labelled corpora — that gives frame-exact labels and direct control over mixing SNR. Splits are disjoint at the source level (speaker id, song, noise file), not the clip level.

Before you deploy this commercially, read this. Two ingredients constrain it:

1. MUSDB18-HQ is CC BY-NC-SA 4.0 — non-commercial research only. It is the only separated-stem source in the pipeline and supplies both halves of singing_music. Whether a model trained on NC-licensed audio is itself encumbered is unsettled and jurisdiction-dependent. Get your own advice.
2. The crawled real-world subset carries no license grant. It went into training only — validation and test stay purely synthetic, so every number above remains comparable with every number measured before and after it was added. Each crawled file has a JSON sidecar recording source URL, video id, uploader, license field, the search query that surfaced it, and a sha256, so any source can be audited and removed on request.

The license: mit tag on this repo covers the code and weights as published by the author; it does not and cannot re-license the upstream training corpora.

Limitations

• 87.95% is an upper bound, not an estimate. Train and test come from the same synthetic mixer and share its biases: no room impulse responses, one SNR distribution, one pool of speakers and songs. Expect real-world numbers below this. Evaluation on genuine recordings (e.g. MIR-1K) is the honest next step and has not been done.
• singing_music is capped by data, not architecture: only 281 distinct sources exist for that class project-wide, because MUSDB18-HQ has 150 tracks.
• Speech coverage is English + Vietnamese. Other languages are untested.
• The 4-second window is fixed by training. Shorter clips are zero-padded; a 4-second window straddling a speech→music cut gets one label for both.
• Music with heavy vocal-like synths, and speech over strongly rhythmic ambience, are the known confusions beyond the singing_music/none pair.

Architecture

TinyAudioCRNN: 4 conv blocks (32→64→128→128, BN + ReLU + max-pool) over an 80-bin log-mel spectrogram, then a 1-layer BiGRU (hidden 128), then mean+max pooling over time into a linear classifier. Per-mel-bin input standardisation (feat_mean / feat_std) ships inside the weights rather than in a config file, so inference physically cannot feed the model a different input scale than it trained on.

Features must match training exactly: 80 mels, n_fft=400 (25 ms), hop_length=160 (10 ms), power=2.0, librosa.power_to_db. The bundled feature extractor does this for you.

Ablations at the same training recipe: transformer (683k params) 82.53%; transformer without label smoothing + SpecAugment 80.13%. So the recipe is worth +2.40 points and the CRNN architecture +5.42.

Selected at epoch 15 of 24 by validation accuracy (90.48%); early-stopped at 23. ~10 minutes on an RTX A4500.

Citation

@software{smad_crnn,
  title  = {SMAD: a small CRNN for speech / music / singing discrimination},
  author = {duclvQ},
  year   = {2026},
  url    = {https://huggingface.co/duclvQ/smad}
}