Two separate things. Sampling randomness goes away with greedy decoding: always take the highest-probability token instead of sampling, which is what temperature 0 approximates.

The remaining nondeterminism comes from floating point. Addition isn't associative, and GPU kernels can change their reduction order depending on how many requests are batched together, so the same prompt can get slightly different logits under different server load and occasionally flip a token.

Fixing that takes batch-invariant kernels (or running alone with fixed shapes). Thinking Machines wrote it up in detail: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...