Not necessarily. Would you be okay with being the dumbest person in the room, not just for one class or one semester, but for four years? A lot of smart kids go to CalTech, MIT and Harvard, but end up dropping out and finishing their degree somewhere else because they're not accustomed to failing on a regular basis. Unless you're convinced that you'll be able to handle repeated failure (because it'll happen when you're trying to achieve at that level) that sort of school may not be for you.
Then there's the fact that these schools are expensive. So not only do you have to be able to deal with failure at an emotional level, but you have to be able to deal with failure financially as well. If you don't have the budget to repeat at least two or three classes - don't bother with a top-ten university.