It took me a long (and often frustrating) time to figure out that the perception of difficulty is largely because of how implicit most everything is in mathematics. It's not like programming where eventually there is a compiler with a definite structure that's going to make your program behave in some exact specific way.
Even worse, is the way we're typically taught mathematics, it's always kinda starting in the middle (as opposed to the beginning). We aren't taught about what it is our how it works, so it's really hard to situate particular mathematical concepts we're taught into a larger coherent framework.
The way I interpret mathematical works changed so much (and for the better), once I realized that any significant bit of math you're using is the result of something a human was trying to accomplish. In the same way you can point to a piece of code and ask it's developer what the motivation was for some abstraction or algorithm, there are always reasons behind the development of mathematical systems. Once I understood that, rather than mathematical systems being these disconnected things that materialized out of nowhere, I started seeing commonalities and considering the reasoning behind them, and seeing more of the network that relates them. It became a lot more enjoyable. I'd recommend checking out some history of mathematics if you'd like to get a better sense of that (my personal intro recommendation: "Men of Mathematics" by E.T. Bell or maybe "Mathematics and the Imagination," which isn't history so much, but serves a similar goal).
If you're looking for something on algorithms specifically the book recommended by the author ("The Algorithm Design Manual") is a pretty friendly read and largely non-mathematical, while still containing all the useful bits (unless you're looking for a reference work on computational complexities or something).