From the start I've always thought that the biggest danger to SO/SE wasn't failure, you guys are smart enough to see and recover from outright failure. Instead it's reaching a level of success far below what you're capable of.
It seems to me that SO/SE spent a lot of work innovating in the early days but once they found their comfort zone they've stopped innovating. They are good sites for their intended purpose but how much of the original vision has been abandoned? I remember back when you and Jeff were talking about taking over the way all online discussion happened. Nothing like that is even remotely possible with the way StackExchange is today.
To be bluntly honest I think you guys are just lucky that most of the ambitious and competent folks in the industry don't view discussion as "sexy" so you don't have as much ompetition as you would otherwise (quora is practically a joke). But realistically it's just a matter of time until SE will be out innovated.