Really. I went to MIT for undergrad, and my experience there was nobody questions who you are in classes with 40+ people. (The one exception was an AI class I took senior year with Patrick Winston. There were almost 100 students, but he quite impressively memorized the names and faces of every student before the first day of class.)

However, you don't get any credit, or a degree. So it doesn't really matter whether the classes you drop in on are Ivy League or not, just that you are learning enough to make it worth your time. You won't be able to validate your knowledge of the course material later with anything but a personal demonstration.

I did this for 3 semesters after running out of money (at MIT, undergrad) and thus being unable to afford tuition. It was a little easier since I was in the system -- I could register, take classes, and then they would deregister me retroactively each semester. I got to take classes, sit exams, etc., just didn't end up with credit.

I actually learned more in classes where I knew I wouldn't successfully remain registered than before, since the only reason was to learn.

If the school itself hadn't screwed me over so much on financial aid/loans (0, since my parents wouldn't pay any of the "expected parental contribution"), I would feel bad for imposing marginal costs.

Or just listen to or watch the classes online. Plenty of great schools have audio or video podcasts of lectures available.