Not exactly what you asked for, but are you aware of CLAP? https://github.com/LAION-AI/CLAP

This generates audio embeddings - much like CLIP does for visual inputs.