1,454 karma · joined September 24, 2015
There is so much that needs to go wrong with our current understanding of math, so that it's a bit like asking "What if we find tachyons that allow us to send messages into the past?" Most likely we will not find them.
If this testing constitutes a criminally negligent behavior, it should be punished. Given the benign outcome (the tip got into spam, Anthropic promptly contacted the police) I doubt that it will make the case.
Correct, but intentions matter. In this case the intention, most likely, was to test a system in the real world environment to catch any anomalies to, in turn, improve the system safety. We don't have enough information to decide whether it was a criminal negligence due to insufficient prior testing of the system in a controlled environment.
We are at a point when you expect that you can safely give a model access, but sometimes the model doesn't do what you expect. Compare this to an advanced driver assist system. You either give it access to your car controls or it's useless. Alive or not the system shouldn't do wrong things (at least, it should be better than a median person in this regard).
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC5830089/ "The observation that hydrolysis of Si NMs proceeds by surface reactions only, in a predictable manner without diffusion through defects or pin-holes"
An LLM mostly deterministically (except parallel processing nondeterminism that can be mitigated) produces a probability distribution that can be sampled deterministically: just take the highest probability token or use beam search.
We'll have plenty of time for this, while living off UBI.
> but training does not produce such LLMs
If we are talking about fundamental limitations it should not be an empirical observation: "does not produce" (which is factually wrong, BTW). It should be a fundamental limitation: "can not produce in principle."
I still don't understand what you are talking about when you say "category mistake." I was talking about computational capabilities of LLMs with CoT that their training can exploit, not about Befunge-98.
What do you mean exactly? "Thinking is not an algorithm."?