Hacker News Reader

Comment by melling | Hacker News Reader

The video says that it's really a logistic curve. Which is also the activation function (sigmoid) in a few machine learning articles that I've been reading.

https://en.wikipedia.org/wiki/Sigmoid_function

https://en.wikipedia.org/wiki/Logistic_function

Can anyone recommend anything to read to add more clarity?

Why does this appear in deep learning too?

View on HN

For neural networks to work, the activation function has to be nonlinear and monotonic. In order to be able to apply the chain rule for training the weights by backpropagation, the function also has to be differentiable. The logistic function is one obvious choice, but other functions such as tanh, arctan or relu are popular as well.

It's used in algorithms to provide a continuously differentiable function, which is much easier analyze than a step function.

In machine learning it's used as an activation function. Activation functions are used to add non-linearity to the model. I think sigmoid functions are inspired from neurons, but I'm not sure exactly how. Nowadays, ReLU activations are used more often than sigmoidals because they are quicker to compute.

>I think sigmoid functions are inspired from neurons I mean the use of sigmoids as activations in NNs

ReLu are mainly used because in practice they show better results than sigmoids in many areas. The fact that they are also faster to compute is just a nice side effect.

> The video says that it's really a logistic curve.

I mean, of course it's not a true, complete exponential, or else it would take over the universe in 2.3 years or something.

All exponentials run out of something eventually. The question is, how soon does that happen.

Reply on news.ycombinator.com