As someone who has worked in deep learning, this isn't true. Figuring out the right recipe is extremely important. The regime in which you can effectively train a neural network with good performance is just very small. If you structure things just slightly wrong, training just won't converge.
IMO we totally could have trained something like a smaller ImageNet with a decently large dataset on a supercomputer in the late 90s. We just didn't know how. In the late 90s and early 2000s, NN researchers didn't know about (or didn't appreciate the importance of) batchnorm, or ReLUs, or which hyper-parameters to use. Convnets had been invented but hadn't yet been popularized.
In the late 90s, you might have tried training an 8-layer deep MLP on a dataset of 50K images on your Cray T3E supercomputer. But you would most likely have failed, because you weren't using convnets, your sigmoid activation function lead to a vanishing gradient, and your hyper-parameters were off. You might let this run overnight, but 12 hours later, your loss barely went down, and you were out of compute budget on a supercomputer you were sharing with other researchers. Given the right code, the right recipe, you could have made it work on that 90s supercomputer and with the compute budget you had, but the right recipe just wasn't known at the time.
We needed time to figure out and to publish about the really basic ingredients needed to train bigger neural networks successfully. Having access to more compute can help figure out what the right recipe is through trial and error, but ReLUs are a super basic innovation. They are a very simple function, and essentially a theoretical innovation. Someone needed to sit down and figure out that vanishing gradients were actually a problem we needed to think about.