Algebra is the simple part. I’d say it’s more about math maturity. At least 1/3rd of my classmates had a hard time grasping the epsilon-delta definition of limit, let alone the deeper definitions like Cauchy sequence or those used in the proof that R is dense(and we were in an elite university’s competitive program). Among the survivors of single-variable calculus, at least 1/3 could barely get by the multi-variable calculus. I saw too many of my friends struggle with different integrals, and got massacred by Green’s equation.

My guess is that most people hit a wall of abstraction at certain point.