Well, one way to think about this is: how much abstract algebra are you keeping in your head to reassure yourself of the security of classical asymmetric cryptography? It's surprisingly deep. Some programmers are "comfortable" with it because they've been brought up being taught that "factoring" is just the way asymmetric cryptography works, but that has never really been the whole case.

Elliptic curve is not at all simple to comprehend! It's easy to implement, but the motivation for designing systems around them (the effectiveness of the index calculus on elliptic curve groups) and the discoveries made on attacking them (like the MOV attack that transforms ECDLP problems to FFDLP problems) are not at all simple.

Arguably, elliptic curve is an odder corner of mathematics than lattices.