The reason you have to optimize SIMD by hand is that compilers can't redesign your data structures and algorithms. This is the level of abstraction at which you often need to be working with SIMD. Compilers are limited to codegen things like auto-vectorizing simple loops, but that isn't where most of the interesting possibilities are with SIMD.

If you look at heavily-optimized SIMD code side-by-side with the equivalent heavily-optimized scalar code, they are often almost entirely unrelated implementations. That's the part compilers can't do.

Note that I use SIMD heavily in a lot of domains that aren't just brute-forcing a lot straightforward numerics. If you are just brute-forcing numerics, that auto-vectorization works pretty well. For example, I have a high-performance I/O scheduler that is almost pure AVX-512 and the compiler can't vectorize any of that.