Is this what softmax is? Simply dividing a vector by sum of its components? If so, then how does it deserve a name, not to mention a long Wikipedia page full of formulas?
1. Transform the components to e^x. This allows the neural network to work with logarithmic probabilities, instead of ordinary probabilities. This turns the common operation of multiplying probabilities into addition, which is far more natural for the linear algebra based structure of neural networks.
2. Normalize their sum to 1, since that's the total probability we need.
One important consequence of this is that bayes' theorem is very natural to such a network, since it's just multiplication of probabilities normalized by the denominator.
The trivial case of a single layer network with softmax activation is equivalent to logistic regression.
The special case of two component softmax is equivalent to sigmoid activation, which is thus popular when there are only two classes. In multi class classification softmax is used if the classes are mutually exclusive and component-wise sigmoid is used if they are independent.
E.g, we have "regression" and "classification" instead of "supervised continuous prediction" and "supervised discrete prediction".
Plot max(0, x) and softmax(0, x) functions, and it should become clear.
Yeah it makes sense that this is a super important function, but I still feel like one could just remember the principle that "exponentiation followed by normalization is a smooth approximation to maxarg."
This opens up possibility of using something else than softmax in there.
If you just say "normalization layer" how am I supposed to know which normalization you're talking about?