If you don't trust the people making your OS, your problems are much deeper than fretting about off-device AI processing.
Even if the org has been trustworthy to this point, I think this step makes it more likely (maybe still unlikely, but more likely) that in the future they do look at your data, as less things have to change for that to happen.
Trust can be abused, certainly, but it also allows collaboration and specialisation, and without those I doubt we'd have gotten very far.
Better to trust many people narrowly (e.g. I don't trust the bus driver to drill my cavities) than to trust a small handful of people broadly (e.g. like Apple expects of their users).
That's not to mention the argument that any software of a LOC count of higher than some number is impossible to audit because of complex state handling. Rice's Theorem applies to your brain too, probably, to some extent. Idk about purely functional Haskell though.
What makes you think this?
The GrapheneOS people are doing this, for example. It's not crazy to consider your device vendor as part of your threat model, because like it or not, they are a threat.
I'm focusing on simpler datasets for now, I want people to be able to annotate a paper restaurant menu with notes about allergens in a way that other people with those allergens can summon those annotations and steer clear--all without participation from the restaurant. Like an augmented reality layer for text.
I hope it grows up into something that would let you ask:
> How trusted is this line of code, and by whom?
The choice is either many little trust relationships or a giant leap of faith. I feel better served by a giant leap of faith and access to all the technology I can use.
I prefer no documentation and consistent behavior to no documentation and a bunch of internet howtos on what might work.
That said I really don't disagree with this point at all in terms of it being a valid problem. It's not a fixable problem either (it comes down to, again, building trustworthy computers) but it could be biased way towards being solved whereas today it is still "trust me bro". I don't think Apple will be the company to make progress towards this, though.
If you don't update your local software then it will certainly become automatically backdoored by an accumulating series of security vulnerabilities over time.
> I don't think Apple will be the company to make progress towards this, though.
I agree.
Y'know though, when you put it that way, it sounds inherent that security vulnerabilities will pop up, which is kinda true, at least for the foreseeable future, but to be pedantic, the security vulnerabilities are already there, it's discovering them that's the problem. If we could make secure computers... (time to formally prove everything from the ground up I guess.)
But, that said, I wasn't overlooking this, I'm just looping "getting pwned" into "active involvement". If you have some sufficiently isolated machines, they're probably fine indefinitely. The practicality of this is limited outside of thought experiments. However it's definitely worth noting that unlike a compromised remote, it is at least technically feasible to work on the problem of making local compromise more evident, whereas a remote compromise is truly impossible to reliably be able to detect from the outside.
The eternal dream of unplugging, and living free on Amigas.