What's the highest level of math that you know so well you could walk into any classroom and teach it? For many, it's been so many years since they've had to use anything from even middle school algebra that they might have to start there.
Make sure you're building on a strong foundation if you wish to build something grand that persists.
You want to find the highest level that feels too easy, and then gradually increase difficulty from there.
You can use sites like Khan Academy to find that point. Alternatively, you can just embrace the fact that you may be overconfident and work through books that seem "below your level" just to verify that you really have mastery over the subject matter.
You could spend a year or two going through the Art of Problem Solving books which cover pre-collegiate math at a deeper level and use SRS like Anki to schedule review of concepts and problems you've understood and solved.
At the college level, assuming you're a programmer, the first subjects to master are probably how to write proofs well, calculus in single and many variables (e.g., Spivak followed by Hubbard and Hubbard), discrete mathematics (e.g., Rosen or Epps), linear algebra (e.g., Strang, Axler, Shilov, etc.), and statistics and probability.
Hang out on math forums like the math subreddit and you'll see many books recommended over and over again.
Be sure to examine the preface and introduction of any book you're thinking about buying. They usually include a section on the intended audience and what their background should be to productively engage with the text.
The most important thing is that you solve or attempt to solve many problems, and avoid looking at the solution until you've really spent a good deal of time banging your head against the wall trying to figure it out. How much time you should wait scales with the difficulty of the subject (e.g., less than an hour for high school level material, a week or more for advanced undergraduate level material, etc.).