Per https://ai.google.dev/gemma/docs/embeddinggemma/model_card_2... the audio data mentions speech, environment sounds, and "acoustic events" and for cross-modality "paired text, image, video, and audio data" so I imagine its audio gets encoded mostly to transcribed language and labels like "bells" without rich structural content.
You might want to actually try giant models with native vision (audio optional) input on spectrograms for audio. Depends on what you need embeddings for and if you're able/willing to extract embedding models out of the LLM (some are more or less amenable) but it's actually insane how good can be at "understanding" audio content's spectral profile.
Try giving opus 5.5 audio spectrograms and ask it to tell you what's inside or even to change the notes/words :)