Yale is kind of where you go if you're part of the American establishment but not too bright. The lesser Bush went there. The ones who are both well-connected and smart go to Harvard or Stanford, or maybe Princeton.
"Not too bright" is very relative. Even W Jr. is a genius compared to the average American. Even with legacy considerations Ivys aren't accepting truly incapable people.