If put AI in a sandbox that can't write to disk, can't call OS API, can't use a shell, can't search the internet to ground results, can't download files off the internet, etc it will be much less useful than an AI without such a sandbox. The incentives for users to not use a sandbox in order to get the most utility, but this also means that there is danger from AI misalignment. And just because you are running it locally that doesn't mean there is not a misalignment risk. And just because people notice, that doesn't mean people will stop using models. Take a look at how much outrage people have had over the overt censoring of Anthropic's Fable model, but at the same time they continue to pay Anthropic and use Fable.

The model itself just generates text, but the harness has done more than that since 2023 and is part of the reason on why AI is better today than it was years ago. I do not believe that the general public will want to give all that up just to improve privacy and will end up asking sensitive questions to agents that have access to sending a signal to the outside world.

>it would quickly be noticed.

If for example Microsoft had a local Copilot model integrated with Windows and people noticed this happening. Do you think all Windows users would just stop using Copilot. Or would you expect the internet to get upset, privacy conscious users switching to something else, and for the majority of Copilot users to remain using it.