These are not brains, they are not even models of brains even if they use the word 'neuron'.

You can use gradient descent one example at a time. It still works just fine. The gradients are more unstable, but you will still converge eventually.