Nevergrad: Python toolbox for performing gradient-free optimization

Nevergrad seems more similar to Nelder Mead optimization, except that Nevergrad is a framework and not a single algorithm, and it has an impressive list of optimizers.

Back in the day before autograd optimizers became a commodity, I used Nelder-Mead and the likes heavily. To this day, I still use gradient-free methods for cases where the first derivative is discontinuous (but I suspect that e.g. Pytorch has some black-magic for those cases).

It would, but it would be super inefficient and slow.

For smooth functions, gradient information gives more information about the search terrain. The more exact your gradient information, the higher quality your iterations become.

On the other gradients aren't defined everywhere (e.g. non-smooth functions like abs(x), max(0,x), or discontinuous/discrete integer variables like x=1,2,3,4...), and gradient based methods can land on saddle points. This is where derivative free optimization (DFO) can help. The downside is they converge far, far, far slower than gradient based methods. It's unlike DFO would scale to what is needed to solve even basic DL models.

Gradient free methods work on any optimization problem, including deep networks, because gradient free methods do not require a smooth parametrization of the search space. For deep networks the parameters are on a smooth manifold and that's why gradient based optimizers are used to find near optimal solutions/parameters, i.e. the search space has the structure of a smooth manifold so the parameters can be updated with gradient descent.