I'm late to the party, but what stops training this SD model on audio spectrograms? Then you'd tell it "some mozart-style violin for 5 seconds, add drums in background." The spectrogram is then translated to sound and suddenly you're a very decent music writer.

With img2txt you could give it an audio file, call it "S" and tell "music in S style, but with flute".