Melati-AudioSet-Ensemble

Six AudioSet taggers run on the same clip, their probabilities averaged: 0.5219 mAP on AudioSet's evaluation set, ahead of every single model we scored on the same clips, the best of them our own Melati-AudioSet-EAT-Large (0.5124). Four members are released by others (CED-base, EAT-large, SSLAM and BEATs) and two are Melati-AudioSet-EAT-Large's v3 and v4. It names which of AudioSet's 527 sounds are in a clip, for six forward passes (1.20B parameters in all); Melati-AudioSet-EAT-Large is one model of 309M, distilled from three of these members' average. Part of Melati, Scicom's speech-data pipeline.

All on the same 17,141 clips: AudioSet's evaluation set as agkphysics/AudioSet holds it today (YouTube removes clips over time, so every paper's copy differs and published numbers are not comparable across papers), each model in fp32 with its own preprocessing, mAP the mean over the 527 classes of their average precision. Every number here is this repo's, loaded from the Hub as the Use section loads it. CED-base reads 0.4993 here and 0.4997 on Melati-AudioSet-EAT-Large's card, where it was given 64 clips at a time (see Use); the other numbers there differ from these by 0.0001 at most (that card scores probabilities stored in 16 bits).

Use

pip install torch numpy huggingface_hub safetensors soundfile

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("Scicom-intl/Melati-AudioSet-Ensemble")
sys.path.insert(0, path)
from melati_ensemble import Ensemble, read

model = Ensemble(path)             # the six members, only their weights fetched, each at a pinned revision; the GPU if any

probs = model([read("clip.wav")])[0]    # any sample rate or channels, read as 16 kHz mono; AudioSet's 527 classes
for i in probs.argsort()[::-1][:5]:
    print(model.labels[i], round(float(probs[i]), 3))

A clip is 16 kHz mono, cut or zero-padded to 10 s, the length every member was scored on. A longer recording goes through model.windows(wav, hop_s=5.0), which gives each 10 s window's start in seconds and its probabilities; the highest over the windows tags the whole recording. model(clips, each=True) also returns every member's own probabilities, and Ensemble(path, members=[...]) runs only some of them (their names are in ensemble.json).

Every member is built in melati_ensemble.py in plain PyTorch, and only its config.json and model.safetensors are fetched from its repo at the pinned revision: no repo's own code is run, so it needs neither transformers nor timm nor torchaudio (soundfile only for read). Each member is fed its own preprocessing, as it was scored, computed for the whole batch on the GPU. On 512 evaluation clips, each member's probabilities match its own repo's code to within 3e-05 in fp32. CED-base's decibel floor is set under each clip's own loudest value, as its feature extractor sets it for a clip given alone, so a clip's tags never depend on the clips batched with it. Revisions before this one ran each member's own code through transformers' trust_remote_code, which needed timm and did not load without it.

On one GPU, in batches of 16, it tags 18 ten-second clips a second in fp32; Ensemble(path, bf16=True) tags 65 a second at 0.5219 mAP. Tested with torch 2.8.0, numpy 1.26.4, huggingface_hub 0.36.2 and safetensors 0.8.0, and loaded into an empty cache in a new environment holding only the five packages above, at torch 2.15.0, numpy 2.5.4, huggingface_hub 2.3.0, safetensors 0.8.0 and soundfile 0.14.0.

Inference engine

melati_engine.py serves the ensemble on CUDA GPUs: every GPU holds all six members, their weights in bf16, and runs a CUDA graph captured for each batch size, and the 10 s windows of every request share one queue, so concurrent requests fill the same batches and every GPU takes its share. A recording longer than 10 s is cut into 10 s windows every hop_s seconds (5 by default), and a sound's probability in it is its highest over the windows.

pip install torch numpy huggingface_hub safetensors soundfile fastapi uvicorn
hf download Scicom-intl/Melati-AudioSet-Ensemble --local-dir melati-ensemble
python melati-ensemble/melati_engine.py serve --gpus 0,1 --port 8000
curl -s --data-binary @clip.wav "localhost:8000/tag?top_k=5"
python melati-ensemble/melati_engine.py tag clips/ --gpus 0,1 --out tags.jsonl    # every audio file under clips/

POST /tag takes an audio file as its body (anything soundfile reads), or JSON {"pcm16": base64, "sample_rate": 16000}, and answers the top_k most probable sounds (threshold sets a floor); windows=true adds each window's, members=true each member's and probs=true all 527. In Python, Engine(gpus=[0]).tag(wave) gives each window's start and probabilities. GET /stats says how full its batches are and how busy its GPUs.

Through the engine on 4 GPUs the evaluation set scores 0.5220 mAP (Ensemble(path, bf16=True): 0.5219), at 282 ten-second clips a second, 2,820 times real time and 3.9 times one GPU's 72. One GPU runs about as fast as Ensemble(path, bf16=True) in batches of 64 (69 a second): a clip costs six forward passes, 1.25 TFLOP, so the GPU's compute is the limit, and what the engine adds is the work spread over GPUs and concurrent requests batched together. An idle engine answers a clip in 38 ms (45 ms without CUDA graphs). Over HTTP on 4 GPUs (the server in one process, clients posting 10 s WAV files), one client gets each answer in 40 ms; 16 clients at once get 152 answers a second (88 ms at the median), and 256 at once 201 a second.

What each member adds

How much the ensemble loses without each member, on the same clips. The three released taggers Melati-AudioSet-EAT-Large v4 was distilled from average 0.5166; with v3 and v4 that is 0.5189, with BEATs 0.5201, and with all three 0.5219. It misses BEATs most (+0.0030), the weakest member alone (0.4835): it was pre-trained differently from the rest. v3 and v4 are the same stage of Melati-AudioSet-EAT-Large trained twice, with and without a frame-level loss, so each adds little once the other is in (+0.0004 and +0.0002). EAT-large adds nothing on top of the rest (-0.0001); v3 and v4 were trained from it.

Members

Labels

All 527 of AudioSet's classes, as model.labels names them (the index is the output's column), under the 7 top-level categories of AudioSet's ontology (Google, CC BY-SA 4.0); the 29 classes it places under two categories are listed under the first. AP is this model's average precision on the class over the 17,141 evaluation clips in fp32, and clips is how many of them carry the label; the mAP above is the mean of the AP column.

Human sounds: 72 labels, mean AP 0.513 Animal: 65 labels, mean AP 0.664 Music: 146 labels, mean AP 0.495 Natural sounds: 17 labels, mean AP 0.511 Sounds of things: 166 labels, mean AP 0.551 Source-ambiguous sounds: 40 labels, mean AP 0.400 Channel, environment and background: 21 labels, mean AP 0.310

Licence

melati_ensemble.py and this card are MIT. The member networks in melati_ensemble.py are written after their repos' modeling code (EAT's, MIT; CED's, Apache-2.0), and its preprocessing and resampler after torchaudio's (BSD-2-Clause). BEATs's weights and code are Microsoft's, MIT (melati_beats/LICENSE); the code is unilm's, unchanged but for its imports of itself, made relative, and its torchaudio import, moved into the one method that uses it. The other members' weights are fetched from their own repos under their own licences: Apache-2.0 (CED-base) and MIT (EAT-large, SSLAM, Melati-AudioSet-EAT-Large). AudioSet's labels are CC BY 4.0 (Google); its audio is YouTube's and is not redistributed here.

Citation

@misc{zolkepli2026melati,
  title  = {Melati: Multilingual Expressive speech, Labelled for Affect, Tags and Interjections},
  author = {Husein Zolkepli},
  year   = {2026},
  note   = {Scicom (MSC) Berhad}
}

We thank Scicom (MSC) Berhad for the compute.