Have been dealing with this issue recently. memcpy was quite slow in my tests however, it was ~twice as fast to check the alignment and load as floats (was dealing with floats in my case).
There have been several memcpy performance regressions affecting modern CPUs in recent years, might want to investigate if that's part of what you were observing.