I think part of the problem is that the things 'voice' is good for tend to have humans on the receiving end that then carry out complex, human actions. "Stu, could you schedule a dinner for me at a good local Italian place at 7:30 and invite Scott and Robin?"
Well shoot, that is a lot to unpack! It's a hell of a lot for a computer to unpack. It's a lot for anyone to unpack unless they know you and your circle of contacts. Carrying it out would require interaction with at least 3 people over a variety of mediums.
I suspect that by opening Siri up (And other digital 'assitants') it might promote the growth of infrastructure and services that would begin to make some of these more advanced queries a little more tractable.