When you realize that most deep learning library, "einsum is all you need", and that einsum is just loops and sum, it kind of screams you are doing it the wrong way when using numerical libraries ranging from BLAS to cuda.
Loops can also be automatically differentiated. Loops can also be reordered. Loops can be tiled. There are some techniques (polyhedral loop optimization) to do it automatically. Loops can be parallelized and vectorized.
Most of the optimization are always the same and due to the memory hierarchy. The problem space is well defined. Candidates solutions can be benchmarked automatically. Some architectures like to have their memory access in a Structures of Arrays, other in Array of Structures, or even Arrays of structures of Arrays. Benchmark depend of sizes in various dimensions. Are you really always going to write the optimizations manually, and do the complex code transforms by hand.
This feels like one area ripped for disruption. The other day there was this thread about not writing your own computer language, but here one domain specific language can have a huge impact. Take the naive form and automatically generate the forward, backward, (nth order autodiff) optimized version given an instruction set whether for cpu, gpu or tpu, and for various memory usage profile.
Optimizing code, even though it feels quite fun and rewarding to do manually is a job for compilers.
Write naive algorithms but smart compilers.
Extra kudos if you handle the mathematical function : For example if you want an exponential : write it as an infinite taylor sum. If you want matrix exponential write it as a taylor sum also ( and if your compiler automatically derive the Pade approximation for improved numerical stability you know you are on the right track).