With squared loss where it's easy for the loss to be zero, then yes, it will have lots of global minima, all at a loss of zero. For losses that asympotote, like logloss, they may have no minima.
Even then, you may have still converged on a local optimum which was the take away I got from the article