If you want to learn mathematics but do not have the time for a full time degree do it part time.
You are most definitely not actually going to reach the same level of mastery of mathematics as even the most mediocre graduate by studying alone. It requires a huge amount of commitment and staying focused and discipled enough is very hard when you don't have a set goal and good continuous guidance and motivation.
It is also very boring and much less productive to study alone. I'd estimate that 70% of my learning comes from the tutorial sessions. Reading the literature is just the starting point. It only counts for 10% if that. Homeworks and exams are also very useful because they make sure you have the prerequisites to move on and start learning something new.