How do we know you're actually running the code and it's not just the LLM spitting out what it thinks it would return if you were running code on it?
Does it matter if I answer every question with either 1 or 2 and flip a coin each time to decide which?
Deterministic means that if it is accurate/correct once, it will continue to be in future runs (unless the correct answer changes; a stopped clock is deterministic).
I think the analogy breaks down here. The elided bit "time indicator" implied at the end makes that statement is false. A stopped clock is not a deterministic time indicator.
If the correct answer changes, a (correct and accurate) deterministic model either gets new input and changes the answer accordingly, or is not correct to begin with.
LLMs can be deterministic if you run them with a temperature of 0 or a fixed random seed, and your kernel is built to be deterministic, but they're not typically used that way, and will produce different output for identical input.
I never said it is. That's why I qualified my example with the word correct.
> no matter what you do, it gives you the same output
This is not deterministic. This is determined. I think this is the confusion I was pointing out.
>> Deterministic means that if it is accurate/correct once, it will continue to be in future runs (unless the correct answer changes; a stopped clock is deterministic).
The bit in the parenthesis, I am trying to argue, is nonsense. If the correct answer changes, the system is not accurate or correct to begin with so the point is moot. Correcting the system will make it accurate. A stopped clock is not deterministic, it's determined. As a time indicator, a stopped clock is not a correct, accurate or deterministic model at all under any possible interpretation.
I build a box. It has an LCD display. It has a button labeled “what time is it”. You push the button and it always shows “10:43am”. This is a deterministic system.
Determinism is about the behavior of a system. Correctness is also about the purpose of a system. A system can have deterministic behavior while being completely unfit for its purpose. And depending on its purpose, it can be fit for purpose while being nondeterministic.
Ask it what the capital of France is, and it will tell you it is Paris. Same with "how do I reverse a string in Python", or whatever problem you have at hand that needs solving (sans searching capability, which makes things more complicated).
So does not the problem need to be unique if you want to be able to claim with certainty it indeed has been executed? I am not sure how you account for the searching capability, and I am not excluding the possibility of having access to execution tools, pretty sure they do.