I end up spending large amounts of time figuring out the meaning of all the various symbols in the paper and how I can map that to an actual computer program.
Thankfully I have enough Math in the back of my mind to work through it, but good luck getting in to the really interesting things (e.g. machine learning, 3d graphics, whatever) without having a good mathematical background. Without it, you'll lose first hand access to almost any paper written on the topic.