It's likely because GPU calculations are non-deterministic and small differences in floating point numbers could lead to different outcomes (either in the way you described or somewhere deeper in the model)
Tensorflow is non-deterministic for some operations due to thread scheduling. PyTorch doesn’t have this issue.