Hacker News Reader

Comment by wenc | Hacker News Reader

> make more general purpose code differentiable

Non-smooth functions (e.g. abs(x)) can be handled with bundle methods, but how would one make inherently discontinuous (non-convex) functions differentiable? (e.g. if x then 1 else 5)

Discrete problems are inherently non-differentiable. There are approaches like complementarity methods and switching functions (tanh) usually end up with numerical issues.

View on HN

This already happens to an extent in existing ML pipelines. The ReLU activation function, is discontinuous in its derivative, and is one of the most widely used functions in neural networks. Its derivative looks like this.

if(i<0) return 0; else return 1;

Now ReLU is continuous itself (as well as being monotonic) so it still cooperates relatively well with gradient descent algorithms. I think this is where the problem lies - not with differentiability itself, but with gradient descent not working due to the highly non-convex search space that such general programming constructs will produce.

ReLU is not discontinuous; it is nonsmooth but continuous, hence derivatives exist except at the hinge points.

Inherently discontinuous functions OTOH are disconnected and nonconvex. Gradient descent works, but you have to add a step to first partition the discrete space like branch and bound. This involves solving the continuous relaxation to find a bound. This does not require differentiability (it is not differentiable), but the price to pay is that it is combinatorial (NP hard)

The OP was talking about general differentiability but inherently discontinuous functions form a large and important class of functions (from software programming) that are not differentiable.

I could be wrong, but I think we're saying exactly the same thing. :-) I agree, anyway.

> Now ReLU is continuous itself (as well as being monotonic) so it still cooperates relatively well with gradient descent algorithms.

A function with a discontinuous derivative cannot cooperate with gradient descent algorithms. That's why you have the famous problem of "dead neurons".

Dead neurons do not come about because of ReLU's discontinuous gradient. It is because for large regions of parameter space (i<0) the gradient is 0.

> Dead neurons do not come about because of ReLU's discontinuous gradient. It is because for large regions of parameter space (i<0) the gradient is 0.

If they had a continuous derivative that would not be the case.

Imagine an alternative ReLU which had a narrow curved section, to smooth out the discontinuity. Now it has a continuous derivative, but that gradient is still zero for values < 0. This flat region is the cause of dead neurons, because backprop multiplies the propagated error by the gradient, to update the weight - and if the gradient is 0, the result of the multiplication is 0 , and the neuron's weights do not get adjusted.

A derivative can be continuous, and zero.

And then people go to empirical data and apply the Great Smoothing: By dropping ML/DL methods on the data (also including results from discontinuous behavior), continuity is often implicitly assumed.

I think the concept you might be looking for is "piecewise differentiable".

Or just use non-gradient based optimisation? Maybe I'm missing something.

That's usually really inefficient. But I agree that's the right direction (pun intended).

Question is: How can we extract similar amount of information about the problem without relying on derivatives?

Reply on news.ycombinator.com