Neural networks are absolutely based on a very simplified model of how brains work. Specific NN architectures are in turn based on specific parts of the brain (e.g. Convolution Neural Networks are based on the visual cortices of cats/frogs).
the tenuous connection DL once had to neuroscience (perceptrons) is a distant memory
AlexNet was a state-of-the-art image recognition net for a (relatively) brief amount of time. It wasn't the first CNN to use GPU acceleration, and it was quickly eclipsed in terms of ImageNet performance.
Regardless, I think bringing up AlexNet kinda invalidates your initial point. Although yes, it turns out that the two were a great match, CNNs and modern GPUs were clearly developed independently of each other, as evidenced by the many, many iterations of both before they were combined.