It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
This conceit, that boosters deserve this and that even if they don't there's no way to stop it so civilization must absorb collateral damage, is ridiculous.
2. There’s no mechanism by which “we as a civilization” can make the decision you’re saying we can
3. Even if there were a mechanism for it, there’s no way we would actually use it
Sure there’s no physical requirement the train keeps rolling. But there’s something even worse! A web of incentives that say the train must keep rolling.
IMO the only way this slows down is through capital failure. But that will not be due to a decision to slow down, and will (ironically) be just as immutable by people’s personal desires.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
You're saying "trained to find a solution" like it's a slur. You sir are a meat popsicle that's trained to find the solution to how does DNA successfully replicate itself. If you want to take higher thought out, then they'd be at the level of animals.
We are fucking around with things that are closer to biological complexity than computer program complexity.