LLMs are computation, they are very complex, but they are deterministic. If you run one on the same device, in the same state, with exactly the same input parameters multiple times, you will always get the same result. This is the case for every possible program. Most of the time, we don’t run them with exactly the same input parameters, or we run them on different devices, or some part of the state of the system has changed between runs, which could all potentially result in a different outcome (which, incidentally, is also the case for every possible program).
GPU operations on floating point are generally not deterministic and are subject to the whims of the scheduler
the mathematics might be
but not on a GPU, because floating point numbers are an approximation, and their operations are not commutative
if the GPUs internal scheduler reorders the operations you will get a different outcome
remember GPUs were designed to render quake, where drawing pixels slightly off is imperceptible