larger_clap_music (corrected conversion)

A Transformers ClapModel conversion of LAION's original CLAP music checkpoint lukewys/laion_clap/music_audioset_epoch_15_esc_90.14.pt (SHA-256 fae3e9c087f2909c28a09dc31c8dfcdacbc42ba44c70e972b58c1bd1caf6dedd). It is a drop-in replacement for laion/larger_clap_music: config.json, the preprocessor and tokenizer files are copied unchanged from laion/larger_clap_music@52455da6f6b32b298341c4270923cca47c7876b3; only model.safetensors differs.

Why

The weights published at laion/larger_clap_music do not match LAION's original checkpoint. Only 20 of its 555 tensors (non-learned buffers) equal a conversion of the original; both projection heads have zero biases and initial-scale weights, and logit_scale_a/t are about 0.03 and -0.01. Every input therefore maps to nearly the same vector, which matches discussion #2 (zero-shot scores of about 0.5 for every label).

How it was made

convert_clap.py (in this repository) maps the original state dict onto ClapModel: it renames modules, splits the audio attention qkv into query/key/value and maps the projection MLPs. The original's STFT/mel frontend, attention masks, tscam_conv, head and *_transform weights have no counterpart in ClapModel and are unused at inference; the text_model.embeddings.token_type_ids buffer (zeros) is taken from the original HF file.

The converter was checked on LAION's general checkpoint: converting music_speech_audioset_epoch_15_esc_89.98.pt reproduces all 555 tensors of laion/larger_clap_general@16c8cc3159a3c8a31e8ff5ef1f66b0d9ab3667db exactly.

Check

Six clips (sine, white noise, silence, an electric piano loop, vinyl crackle, a kick drum) against six matching descriptions, unit-normalized embeddings:

This is a sanity check that the model discriminates, not an accuracy benchmark.

License and credit

The original checkpoint is released by LAION under CC0-1.0, and laion/larger_clap_music is Apache-2.0; this conversion is distributed under Apache-2.0. Model by LAION: Wu, Chen, et al., Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation, ICASSP 2023.