Two separate things. Sampling randomness goes away with greedy decoding: always take the highest-probability token instead of sampling, which is what temperature 0 approximates.

The remaining nondeterminism comes from floating point. Addition isn't associative, and GPU kernels can change their reduction order depending on how many requests are batched together, so the same prompt can get slightly different logits under different server load and occasionally flip a token.

Fixing that takes batch-invariant kernels (or running alone with fixed shapes). Thinking Machines wrote it up in detail: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Choosing the next token deterministically.

The question should really be the other way around - LLMs are just computer programs, and in general, computer programs are theoretically deterministic. Given the same inputs and the same initial state, they produce the same outputs. So the question is, what would make an LLM non-deterministic?

For any computer program, there are two main ways determinism can be lost:

1. Randomization is introduced deliberately, e.g. via random number generators.

2. The program's behavior depends on factors beyond its explicit inputs, e.g. its execution environment, timing, concurrency, or interactions with other systems.

Actual deployed LLMs typically lose determinism because of both of the above factors:

1. On each request, the core model outputs a list of likely tokens, ranked in probability order. To avoid repetitiveness or getting "stuck", particularly in natural language output, it's common to introduce randomization when picking a token from this list. This is what's called "temperature". This is deliberate non-determinism that's applied to the output of the core neural network model. Setting temperature to zero can eliminate this kind of non-determinism.

2. In practice, consumer-facing deployments run in complex hardware environments - distributed processing, GPUs, hardware shared with other processes, etc. This leads to non-determinism due to things like concurrency and timing issues. Calculations can execute in different orders, numerical errors can accumulate in different ways, etc.

For companies like OpenAI or Anthropic, which serve millions of concurrent sessions, eliminating non-determinism from #2 would be difficult, and not worth the effort.

But if you're running an LLM on e.g. a single machine, it's certainly possible to eliminate non-determinism. People do this for various kinds of models. Machine learning frameworks have explicit options to help with this. E.g. with Pytorch, you can use `use_deterministic_algorithms(True)` to force it to use deterministic algorithms where possible.

One point that sometimes confuses people here is that even if you run a perfectly deterministic LLM, the output is unpredictable in general. That's because the output of the model depends on a large number of trained weights, and the only practical way to "predict" what output a model will produce is to run input through the model. This is not non-determinism, though.

In diagram terms, at a high level an LLM looks like this: input --> neural network --> token selection --> output

Token selection can be randomized, but doesn't have to be. When it's not, the above is perfectly deterministic in theory, and can also be run that way in practice.

Now take that diagram and run it across a data center full of machines, and achieving determinism gets more difficult. It could be done if we wanted to, but in general it's not important enough to try in that scenario.

Being more strict with how you perform operations and then using a set seed. It isn't super useful in most cases.