Hacker News Reader

Comment by monster_truck | Hacker News Reader

They're actually extremely predictable. What you are experiencing is the provider you are entrusting with your time, effort, and money fucking you because they need the capacity to train the next one. So without any real notice or indication other than things vaguely seeming broken in ways everyone seems to be experiencing differently, they're serving up some shitty quant of what you already paid to use.

View on HN

Extremely predictable is an exaggeration. I was teaching machine learning and we needed half a dozen lines to make even simple networks deterministic. LLMs are not really predictable, even if we want to, they will sometimes behave almost chaotic depending on the input.

I'm not sure how you're arriving at "if you use the same seed with the same version of the software on the same hardware with the same drivers you get the same result is an exaggeration", it's wild that you're a teacher and you seemingly do not know this.

Order of operations affects the result in floating point, even moreso at lower precision. GPUs are massively parallel devices with many different sources of latency, so order of operations is not guaranteed for things like dot products, where many threads contribute to the result. Consequently, you can get non-deterministic outputs even though nothing else has changed.

> we needed half a dozen lines to make

6 lines per model? Can't you put that 6 lines in a function/macro/decorator/library so you only have to call it?

My understanding of LLMs are that they are giant functions and generate output deterministically. However, the way these providers setup and route traffic means that your traffic lands on different boxes where each box has different inputs that run alongside your prompt -- essentially creating "non-deterministic" behavior. Basically it's BS to make LLMs look sentient. Each model could be burned onto an ASIC if you really wanted to be locked into that model and get incredible token speeds. If I'm wrong I would like to be corrected, thanks.

I replied to your neighbor https://news.ycombinator.com/item?id=50432889

Reply on news.ycombinator.com