I mean they can be deterministic, thinking machines lab made deterministic LLM's: https://thinkingmachines.ai/blog/defeating-nondeterminism-in....

Although obviously I'd rather they stay non-deterministic