I've experienced what you're talking about as maybe the fundamental challenge in making computer-based art. Computers love to follow rules. Come up with some interesting rules, and they will spit out interesting pictures/music. It's much much harder to figure out how to tell the computer "okay, but break the rule just a little bit, at the appropriate time". Codifying breaking the rules as its own rule is...well, so far it's mostly eluded me.