CLAP HTSAT-fused — ExecuTorch (search sound with words)

ExecuTorch builds of laion/clap-htsat-fused, exported with torch.export and lowered to XNNPACK and Core ML. On a Mac M4 Max the Core ML audio tower takes 8.6 ms per 10-second window and the Core ML text tower 4.8 ms per phrase (2026-11-01).

CLIP for audio. An audio tower and a text tower put a clip and a phrase in the same 512-dimensional space, so "a dog barking" can be matched against a recording with a cosine — with words the app chooses at runtime, not a label list fixed at training time.

This collection already has AST, which answers with AudioSet's 527 labels, and CLIP and SigLIP2, which do this for pictures. This is the piece that was missing: a searchable representation of sound.

audio, global: input_features[:, 0:1] (1,1,1001,64) -> (1,512)    clips of 10 s or less
audio, fusion: input_features (1,4,1001,64) -> (1,512)            clips longer than 10 s
text         : (input_ids (1,32), attention_mask (1,32)) -> (1,512)

Both embeddings come out L2-normalised, so a cosine is a dot product. The text side runs once per phrase and can be cached; the audio side runs once per clip.

For clips of 10 s or less, use the global audio files. They take the branch that laion_clap, the reference implementation, takes for such clips. The fusion files take the branch the Hugging Face port takes for a single clip, and the one both take for a clip longer than 10 s (a case not measured here); see Which is_longer branch.

The int8 pair (global audio + text) is 272 MB against fp32's 615 MB, and both keep all five winning phrases. The audio tower quantises well (0.9997); the text tower is the one to watch at 0.9908, which clears the 0.99 bar these conversions use for embeddings but not by much.

*Mac M4 Max, single process — a reference point for relative cost, not a device number. Median of 30 runs, the audio files interleaved in one process and the text files in another (2026-11-01). XNNPACK delegate coverage (fp32 files): global audio 63.8%, fusion audio 64.2%, text 77.3%. Core ML takes the whole global audio graph and the whole text graph.

The text tower is the big one — it is a RoBERTa base, 124.65 M parameters against the audio tower's 27.55 M — and it is also the one you can run once and keep. A phrase set of twenty labels is twenty 512-float vectors.

Core ML for the global audio file only. The audio export used to keep the is_longer index in the graph — a torch.where whose output size is only known at run time — and Core ML stopped on it (Could not extract specialized integer from data-dependent expression u12). The global graph does not compute that index, and Core ML takes all of it. The fusion files are the earlier export, index included, so they have no Core ML build. An earlier version of this card put the stop in the HTSAT windowing; in the global graph it was this index.

Running it

1. The audio. ClapFeatureExtractor at 48 kHz mono: 64 mel bins, a 10-second window, and the four-chunk layout the -fused checkpoint uses — input_features comes out [1, 4, 1001, 64].

• Global file: pass input_features[:, 0:1]. For a clip of 10 s or less the extractor repeat-pads it to 10 s and the four chunks are the same mel.
• Fusion file: pass all four. For a clip longer than 10 s they are the whole clip shrunk to the window plus three crops.

The extractor also returns is_longer. Neither graph takes it: each file is one branch, and the clip length picks the file — at 48 kHz, len(waveform) > 480000 means the fusion file.

2. The phrase. The repo's RoBERTa tokenizer with padding="max_length", max_length=32. Padding length does not change the embedding — checked, cosine 1.0 against padding to the natural length — but the graph's shape is fixed at 32.

3. The match. Dot the two vectors. Both are already unit length, so the dot product is the cosine. For a small label set, embed every phrase once, stack them, and take the argmax.

Which is_longer branch

The -fused checkpoint's patch embedding has two branches, picked by is_longer: a global path over the whole 10-second window, and a fusion path that adds three local crops. torch.export cannot keep the choice (it is a Python if on the size of a torch.where result: GuardOnDataDependentSymNode ... u0), so each file takes one branch.

laion_clap takes the global branch for a clip of 10 s or less. Its inference (CLAP_Module.get_audio_embedding_from_data) repeat-pads the clip to 480,000 samples, stacks its mel four times and sets longer=False; in eval mode its HTSAT then runs the global path only. Only longer clips take the fusion branch.

The Hugging Face port takes the fusion branch for a single clip. ClapFeatureExtractor ends with this:

if truncation == "fusion" and sum(is_longer) == 0:
    # if no audio is longer than 10s, then randomly select one audio to be longer
    rand_idx = np.random.randint(0, len(input_mel))
    is_longer[rand_idx] = True

For a batch of one, that marks every clip is_longer=True, however short. laion_clap has the same rule, but only while training. b4ph showed that the reference takes the other branch, and measured what the fusion branch costs on retrieval: clap-retrieval-benchmark, filed as transformers#50097.

Measured against laion_clap's own pipeline — its repeat-padding, its mel, its model in eval mode — with this checkpoint's weights loaded into laion_clap's model, on the five sounds below:

The table is the fp32 files; int8 and Core ML are within 0.006 of them. Given laion_clap's own mel, the global path reproduces its embedding at cosine 1.000000 on all five. What is left of the gap is laion_clap's input step: it truncates the waveform to int16, which zeroes the quietest samples. With the same step on the HF side the two mels agree to within 0.01 dB; without it they differ by up to 59 dB in near-silent frames.

The weights matter here. Neither public laion_clap fusion checkpoint (630k-fusion-best.pt, 630k-audioset-fusion-best.pt) holds the weights of laion/clap-htsat-fused; b4ph's benchmark found the same for laion_clap's default one. So the reference above is laion_clap's code with this checkpoint's weights, mapped through transformers' own conversion table in reverse. Text embeddings then match at cosine 1.000000.

An earlier version of this card called the global build wrong. That comparison put the two branches against each other (cosine 0.78–0.87), which shows only that they differ.

To compare either file with the stock HF model, set is_longer yourself. With a batch of more than one and no clip over 10 s, the extractor marks one clip per batch at random, unseeded, so which clips take which branch changes between runs. For a clip longer than 10 s, the three crops are also drawn at random, unseeded, in both the extractor and laion_clap, so a long clip's fusion embedding changes between runs too.

What is measured

Two numbers, because an embedding model has two things that can go wrong.

Fidelity. Cosine between each build's embedding and the fp32 eager model on the same branch, minimum over five sounds and six phrases — the table above.

Behaviour. Whether the phrase that wins for each sound is still the same as in eager: 5/5 for every build. Against laion_clap's pipeline both audio files agree on 4/5: the global file differs on Ping, where laion_clap's top two phrases are 0.007 apart, and the fusion file on Glass.

            laion_clap            global file           fusion file
Ping        an electronic beep    a bubble popping      an electronic beep
Submarine   an electronic beep    an electronic beep    an electronic beep
Purr        a bubble popping      a bubble popping      a bubble popping
Sosumi      an electronic beep    an electronic beep    an electronic beep
Glass       rain falling          rain falling          a bell ringing

python scripts/compare_laion_clap.py hf   # the laion_clap comparison; macOS, needs laion_clap (see the script)

The audio is macOS's own system sounds — short real recordings. That matters: the first probe used synthetic speech from say, and the fp32 model rated it "rain falling" at 0.82 before any conversion was involved. CLAP is trained on environmental audio and speech is not what it is for. Five sounds and six phrases is a conversion check, not an evaluation of the model; for retrieval quality, see b4ph's benchmark above.

• Source: laion/clap-htsat-fused
• License: Apache-2.0

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)