> Given the right code, the right recipe, you could have made it work on that 90s supercomputer and with the compute budget you had, but the right recipe just wasn't known at the time.
We could expect the best of the best given access to these powerful computing machines, surely they would know their lenet (1989) http://karpathy.github.io/2022/03/14/lecun1989/ and could have studied statistics of gradients and activations to implement something like ReLu - which is one line of code.It was a low-hanging fruit.
> Someone needed to sit down and figure out that vanishing gradients were actually a problem we needed to think about.
If Sepp Hochreiter could do it, surely someone from CalTech could do it as well, given the caliber of people who study there. Again, looks like a simple problem of misapplication of best and brightest (who, surely, know what they want to work on in life, but are still influenced by academic prestige and their advisors' advice which would repel them from connectionism then).