At a good college you learn as much from the other students as from your classes, and not only about CS.
It's also easier to work on side projects in college.
You should not leave college as early as this unless (a) you start something so successful that it pulls you out, or (b) you are one of the very rare people who are temperamentally unsuited to it.
Unless your college is awful, your longing for SV is a case of the grass being greener on the other side of the fence. SV is full of people who wish they were back in college.