Eigen was my baseline. As far as I remember, my best AVX version (which unrolls 24x4 blocks, running 6 iterations of the loop) was about 50% faster.

Eigen is designed in the assumption these vectors/matrices are large. 24 is not large, fits in 3 vector registers out of the 16 available. Eliminating data access latency of the accumulators is what caused the speedup.