I would say the main reason it's hard to follow abstract maths lectures by listening is not the difficulty of the concepts per se, but simply because it's so visual: it relies on notation and diagrams
But mathematicians can talk to each other about arbitrarily abstract concepts, as long as they have enough shared background, and they don't (always) need a blackboard to do it.
Conversely, you can have conceptually very simple things that are basically impossible to follow just by listening, like multiplying two nine-digit numbers or following one of Euclid's proofs in plane geometry. The difficulty isn't about abstraction, but how many things you have to hold in working memory