Transformer Explainer
poloclub.github.io
poloclub.github.io
> We then apply the softmax function to convert the logits into a probability distribution that sums to one.
To go from a probability distribution to a single token, you need to sample it, ie pick a random element according to the probability distribution[1]. Elements with higher probability have a higher chance to get picked, but it's still random. This is also why the "temperature" parameter works the way it does.
[1]: https://en.wikipedia.org/wiki/Non-uniform_random_variate_gen...
[1] https://en.wikipedia.org/wiki/Principle_of_compositionality