As an example, while doing code for a 6809-based video game (see [1] for full war story), I had to use every single register on the processor, including the system-stack register and the direct-page register, to speed up a graphics routine. This meant that, while my code was running, the system effectively had no stack and couldn't use certain addressing modes. Further, it also meant that interrupts would corrupt whatever memory the stack register happened to be pointing to.
All these problems were solvable – and the pain we had to go through to solve them was, for us, absolutely worth the 70% speed-up it bought us – but no compiler writer is going to incorporate tactics this weird into a compiler's optimization logic. If you need to go "full weird," assembly is often the only way.
[1] http://blog.moertel.com/posts/2013-12-14-great-old-timey-gam...
On this specific case it was developer helping the gcc optimizer. These type of optimizations could produce worse code in another C compiler.
Sometimes the code is already fast enough for the problem being solved, other times rewriting the algorithm could improve performance.
Only in cases where every ms counts, does helping the compiler really matter.
But the 1733.4 MFLOPS solution is a really naive one. Data locality and cache friendliness are two of the first things taught in high performance classes, with an obvious trick : iterating using row-major order like in the second case.
This trivial optimization gives a 9419.8 MFLOPS throughput. I don't think "everyday code" like that is so inefficient given the maximum result which can be obtained (17985.4 MFLOPS).
Relative to the amount of code in the world there are vanishingly few loops which have this kind of 10x speedup just waiting for the right tuning.