Haven’t used Eigen for quite a while, but I do remember it being rather slow to compile once it proliferates. Those templates are clever but also quite taxing.
Even so, using another math library would have been the solution then.
The bits of eigen that make it slow to compile are the same bits that make it both easy to use and fast. I'm sure there are improvements that could be made, but ultimately eigen is big and slow to compile for a reason.