There are lots of reasons, some of which never go away (peer acceptance, additional scrutiny / skepticism, raise availability and amount.) There's also the specialization problem: you can "be fine" but essentially be limited to what I like to term "survival tech" jobs (CRUD and some other less-than-glamorous roles.) Not going to school, especially one that gives you opportunities to work on really interesting stuff (elite/research schools), means you'll likely never break into things like robotics or Socher-level DL. There are exceptions, but most people will not.

You may also never be fully accepted culturally, since one of the major advantages of going is that you're the "right" sort of person (meaning that you have the correct socio-economic background and understand/speak corporate-derived "elite meritocratic culture") / you're able to build a high-quality network of well-connected peers who themselves are part of the socio-economic elite.