There's of course Roger Penrose's "The road to reality" which has a completely different emphasis (more focused on fundamental physics and therefore introduces a lot of the math used there) but is also very interesting to read. This however, is a real time -- will take some effort to go through, but probably worth it :-)
I started with Spivak & Apostol but reverted to Riley and Hobson's Foundations text because I was struggling too much with the A&S's problems. Given that I did very well on symbolic-logic proofs in an undergrad course, I figured it would be easy enough to get into math if I had diligence and sincere interest. After having been a 'D' student, my recognition of arithmetic and algebraic expressions and manipulations is hopelessly sub-par. Even many of the R&H problems are beyond my grasp.
My approach has been informed by a sincere desire to engage with mathematics (one I unfortunately did not have through my formal education) as well as several "How do I self-teach maths" threads on HN, Reddit, and /sci/. Previously I had chalked up my poor grades to an undisciplined, unmotivated youth. Now I'm beginning to suspect I may simply not have the requisite intelligence, or am at least outside the age where I had enough time and neuroplasticity to pick math up in earnest.
So, it's certainly possible, but also kind of not. It depends where you are in your life, and why you want to do that. If you have a job, maybe a family, and think you'd like to invest one-two hours of day for this hobby, I'm not sure you can get there. If you are willing to invest substantially more time and effort, you need to ask yourself why? Do you want to enhance your career? I think there are better areas of math/CS/Machine Learning that you can attack, with a much better return on investment. Do you already have enough money, that career advancement is not a concern, and you simply want to pursue truth and beauty, and you find advanced math to fit your taste? Then you can go solo, and I think you can succeed, but I think it's much more efficient to actually get feedback from other people (via tutoring or attending courses, or even MOOC).
Anyway, not sure if you know about 3brown1blue [2]. Check it out, I hope you'll enjoy it.
[1] https://www.amazon.com/Riemannian-Geometry-Manfredo-Perdigao...
It is possible to learn the math through self study.