This reminds me a great deal of Supercollider, which is pretty robust and time-tested.
One of the things that bothers me about Supercollider is its weird server architecture and esoteric language. I'll take Ruby over that, but these things really beg for APIs that let you hook in with other languages.