The difference is that you come from a background where you see people (eg your parents) earning a good income and believe that a $100k loan is something you can actually afford in the long term. Someone from a poor background whose parents have never earned a high income would very likely see $100k debt as an impossible barrier they'll never get past, and consequently would be a lot less likely to take on the risk.
If schools didn't "soak the rich" then the potential of the student from a poor background would never be realised, and humanity would be the worse for it.