Hacker News Reader

It's close but not fully deterministic. I remember seeing a theory that if the system is considering multiple possible completions that have the same rounded score percentage, then its choice between them is nondeterministic.

View on HN

It's likely because GPU calculations are non-deterministic and small differences in floating point numbers could lead to different outcomes (either in the way you described or somewhere deeper in the model)

> GPU calculations are non-deterministic

Tensorflow is non-deterministic for some operations due to thread scheduling. PyTorch doesn’t have this issue.

Some of the underlying CuDNN algorithms have nondeterministic implementations, which applies to PyTorch as well. See https://pytorch.org/docs/stable/notes/randomness.html

Reply on news.ycombinator.com