I've had some luck with that approach - giving a frontier LLM a reference sound, asking it to replicate the sound via code (basically asking it to create a physical model, I guess) and being able to steer the output by talking with the LLM.

I was working with very simple elements, but I was surprised by some of the outputs.

I would have also suggested Waves Illugen, but it turns out that is text-only, you can't give it an audio reference.