While I think you raise important points about the dominance of hardware optimizations, I think you're massively overstating the simplicity of the algorithms.

Sure, it's easy to code the forward pass of a fully connected neural network, but writing code to train a useful modern architecture is a very different endeavor.