Melati-AudioSet-EAT-Large

An AudioSet tagger: one model that names which of AudioSet's 527 sounds are in a clip, scoring 0.5152 mAP on AudioSet's evaluation set, ahead of every released single model we scored on the same clips (the best of the others is CED-base, 0.4997), our own earlier releases included (v4 0.5123, v3 0.5105). It is EAT-large (309M parameters) distilled from the average of three released taggers (CED-base + EAT-large + SSLAM; 0.5166 averaged) and, in its last stage, of those three with BEATs and its own v3 and v4 (0.5218 averaged, released as Melati-AudioSet-Ensemble) on 730,841 AudioSet clips (2,011 hours), none of them from its evaluation set. Part of Melati, Scicom's speech-data pipeline.

All on the same 17,141 clips: AudioSet's evaluation set as agkphysics/AudioSet holds it today (YouTube removes clips over time, so every paper's copy differs and published numbers are not comparable across papers), each model in fp32 with its own preprocessing, mAP the mean over the 527 classes of their average precision.

Use

import torch, torchaudio, soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained("Scicom-intl/Melati-AudioSet-EAT-Large", trust_remote_code=True).eval()

wav, sr = sf.read("clip.wav", dtype="float32", always_2d=True)
wav = torch.from_numpy(wav.mean(1))
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)
wav = wav - wav.mean()
fbank = torchaudio.compliance.kaldi.fbank(wav[None], htk_compat=True, sample_frequency=16000, use_energy=False,
                                          window_type="hanning", num_mel_bins=128, dither=0.0, frame_shift=10)
fbank = torch.nn.functional.pad(fbank, (0, 0, 0, max(0, 1024 - len(fbank))))[:1024]     # 10.24 s
x = ((fbank + 4.268) / (4.569 * 2))[None, None]                                          # [1, 1, 1024, 128]

with torch.no_grad():
    probs = torch.sigmoid(model(x))[0]
for i in probs.topk(5).indices.tolist():
    print(model.config.id2label[i], round(probs[i].item(), 3))

The input is EAT's: 16 kHz mono, 128-bin Kaldi filterbanks every 10 ms, 1024 frames (a longer clip in 10 s windows).

Labels

All 527 of AudioSet's classes, as model.config.id2label names them (the index is the output's column), under the 7 top-level categories of AudioSet's ontology (Google, CC BY-SA 4.0); the 29 classes it places under two categories are listed under the first. AP is this model's average precision on the class over the 17,141 evaluation clips in fp32, and clips is how many of them carry the label; the mAP above is the mean of the AP column.

Human sounds: 72 labels, mean AP 0.507 Animal: 65 labels, mean AP 0.657 Music: 146 labels, mean AP 0.490 Natural sounds: 17 labels, mean AP 0.509 Sounds of things: 166 labels, mean AP 0.543 Source-ambiguous sounds: 40 labels, mean AP 0.392 Channel, environment and background: 21 labels, mean AP 0.304

How it was trained

The student is trained to the average of the teachers' probabilities (binary cross-entropy against soft targets; AudioSet's own labels are not used) on AudioSet's strongly labelled training clips (102,123, every event timed by hand, from CLAPv2/audioset_strong), its balanced set (11,397) and 617,321 clips of its unbalanced set, any clip of the evaluation set left out, in 4 stages:

1. From EAT-large fine-tuned on AudioSet-2M (0.4942 on these clips in bf16), 10 epochs on the strongly labelled set and the balanced set (113,520 clips, 311 hours), learning rate 1e-5: 0.5027 at its best (bf16).
2. From stage 1, 3 epochs on the strongly labelled set, the balanced set and 617,321 clips of the unbalanced set (730,841 clips, 2,011 hours), learning rate 1e-5: 0.5054 at its best (bf16).
3. From stage 2, 5 epochs on the strongly labelled set and the balanced set (113,520 clips, 311 hours), learning rate 1e-5: 0.5118 at its best (bf16).
4. From stage 3, 3 epochs on the strongly labelled set, the balanced set and 617,321 clips of the unbalanced set (730,841 clips, 2,011 hours), with BEATs and its own v3 and v4 added to the teachers, learning rate 1e-5: 0.5147 at its best (bf16).

Every stage: SpecAugment, warm-up and cosine decay, bf16 with gradient checkpointing, and an exponential moving average of the weights (0.9995) whose best evaluation is kept; stages 1 and 2 also mixed half their batches (mixup). The code is audioset/ in Melati.

Limitations

• One evaluation set: AudioSet's own. On speech-heavy material (podcasts) it tags what AudioSet tags, which is not a measure of how well it times or places an event inside a long recording.
• Clip-level: a probability for each class over the whole input, not where in the clip each sound is.
• Its teachers' average (0.5218) bounds what distilling from them can reach; it is 0.0066 under it.
• The training clips are 730,841 of AudioSet's 2 million; more of its unbalanced set is the obvious lever.

Licence

MIT, as EAT's code and weights it starts from. The teachers it learned from are released under Apache-2.0 (CED-base) and MIT (EAT-large, SSLAM, BEATs); two of them were its own earlier versions. AudioSet's labels are CC BY 4.0 (Google); its audio is YouTube's and is not redistributed here.

Citation

@misc{zolkepli2026melati,
  title  = {Melati: Multilingual Expressive speech, Labelled for Affect, Tags and Interjections},
  author = {Husein Zolkepli},
  year   = {2026},
  note   = {Scicom (MSC) Berhad}
}

We thank Scicom (MSC) Berhad for the compute.