Learning advanced mathematics without going to university would take an extreme amount of dedication, focus, and effort, but it's certainly possible. It's much easier with the resources available on the internet, and being able to connect with people through forums and stack exchange.

John Baez's recommendations: http://math.ucr.edu/home/baez/books.html

For theoretical physics 't Hooft's recommendations: http://www.staff.science.uu.nl/~gadda001/goodtheorist/