I had a much better time with "What is Mathematics?" (aka "Courant and Robbins"), though it still suffers from some similar problems (I remember being baffled by all the attention paid to these properties like commutativity, associativity, etc.—"why should I care about this!?").
Really, I still can't think of a single book that was great for overcoming my own "math illiteracy," it was mostly just a haphazard process of trying enough different things that eventually various parts started falling into place etc.
But I guess part of what's going on is that what works for one person is gonna be awful for another. So I just want to say, maybe "How to Prove It" will be great for you (other readers)—but don't feel bad if it isn't.
I remember reading a few sections from What is Mathematics too. Some chapters on topology and the preface.
So, yeah, I also succeeded because I drowned myself in everything possible until I got used to it. It's just that I never felt any frustrations with Velleman's book, so I knew that every time I open it I will learn something new, won't get confused, and the level of difficulty will be just right. That's why I kept coming back to this book and completed most of it, while didn't make much progress in others.
Also, interesting how much of a similar way we seem to have gone about learning math. I kept it as my main focus for a similar amount of time, then for as a secondary focus while working on a big software side project for another 7 months or so (I was working at a grocery store for money the whole time), then got my first software engineering job and largely dropped any focused study in mathematics. But now I'm in a position where I'll learn bits of math as I need them for projects or whatever, so it definitely wasn't a waste (and like you mentioned in being able to read stuff off of arxiv, it expanded the range of ideas I can understand).
I've actually started getting interested in continuing study again, getting more into applied math this time since I really focused on pure last time. First goal is to get a clear understanding of Maxwell's Equations :)
It took me a long (and often frustrating) time to figure out that the perception of difficulty is largely because of how implicit most everything is in mathematics. It's not like programming where eventually there is a compiler with a definite structure that's going to make your program behave in some exact specific way.
Even worse, is the way we're typically taught mathematics, it's always kinda starting in the middle (as opposed to the beginning). We aren't taught about what it is our how it works, so it's really hard to situate particular mathematical concepts we're taught into a larger coherent framework.
The way I interpret mathematical works changed so much (and for the better), once I realized that any significant bit of math you're using is the result of something a human was trying to accomplish. In the same way you can point to a piece of code and ask it's developer what the motivation was for some abstraction or algorithm, there are always reasons behind the development of mathematical systems. Once I understood that, rather than mathematical systems being these disconnected things that materialized out of nowhere, I started seeing commonalities and considering the reasoning behind them, and seeing more of the network that relates them. It became a lot more enjoyable. I'd recommend checking out some history of mathematics if you'd like to get a better sense of that (my personal intro recommendation: "Men of Mathematics" by E.T. Bell or maybe "Mathematics and the Imagination," which isn't history so much, but serves a similar goal).
If you're looking for something on algorithms specifically the book recommended by the author ("The Algorithm Design Manual") is a pretty friendly read and largely non-mathematical, while still containing all the useful bits (unless you're looking for a reference work on computational complexities or something).
Which is a completely wrong impression due to how it is taught in school and to undergrads in the US.
Speaking as a mathematician, since one is reasoning about abstract objects everything necessarily is pedantically explicit (otherwise proofs could not possibly work). Hence, in US graduate courses or EU undergrad courses you start from scratch and rigorously define every symbol you ever write and justify every step you take (e.g. given a field, why is "1" distinct from "0"? What is a derivative? Prove that it is actually well-defined, exists for such functions etc.).
What many people confuse for being implicit is the heavy polymorphism and terseness in mathematical notation. For instance, one often identifies a function, its graph or its image depending on type makes sense in context. Here, rigorous courses spent much time to prove that any possible ambiguity in notation is actually no ambiguity at all; one is allowed to use short-hand notation because all interpretations are equivalent.
My biggest frustration learning math when I was younger was not knowing -why- I was doing something a certain way or how it worked. It was incredibly discouraging because it didn’t just “click” like it seemed to with my peers.
It's a very healthy mindset to have, but it's a blessing and a curse. Some people are just very good at just abstracting away the details and getting on with things; what seems like "clicking" is not always the same as intuition. Probably it's good to operate in both modes. It's a really really useful skill to know when you need to care about the internals and when you can safely ignore them.
Arbitrary example:
One teacher kept saying "f(x)" but couldn't explain what a function is. He just said "it's anything", then "don't worry about it". If he had even said "a function is like a machine that takes number(s) as input, changes them with a formula and outputs the new number(s)", I think it would have helped us grok.
I think he understood math(s) so well that he couldn't relate to someone who didn't know what a function was.
Helpful but not quite right and one of the common misconceptions students seem to carry over from high school.
A function is something that takes inputs out of a set (its domain, e.g. people in this class) and gives you exactly one output with a prescribed data type (e.g. a date. The function than for instance being person -> birthdate). Neither numbers nor a formula are needed.
> I think he understood math(s) so well that he couldn't relate to someone who didn't know what a function was.
I think he probably could, but the syllabus said explicitly not discuss this ("too abstract") and there is time pressure. High school maths mostly is somewhat handwavy and stringing "definitions" together by examples. So you would mostly see lots of examples of functions and the "definition" of 'function' is then simply "things like that".
I was the same as how you describe yourself, and I always just assumed the kids who were best in the class were doing what I was trying to do also, but were much better at it. And then you see later how fragile/limited one's understanding is when taking the memorization approach—but as a kid you don't have many options.
Schools tend to not teach mathematics but memorization, unmotivated formulas and computation. Mathematics as its own discipline is basically only "why and how", i.e. proofs.
Advanced math is linked with computing mostly because the roots of computing lie in mathematical logic.
So you can become a master programmer without advanced math knowledge.
In short, most algorithms 101 books should be accessible to someone who considers themselves math illiterate.
Statistics, for example, appears in any non-trivial computer science problem, like network congestion, analyzing proper parameters for algorithms, probabilities of getting lock contention, distribution of dispatched instructions, audio psychoacoustics, etc.
Computer vision and graphics programming requires a solid foundation of linear algebra and geometry.
In short, those tools are pretty necessary if you want to approach computer science as a science and as a tool for engineering beyond the very basics.
A lot of real world problems can be solved without a programmer requiring advanced math knowledge.