For the AI/OS intersection, it is indeed a very interesting design space. The key insight really is that the better the AI knows you, the more helpful it can be to you, so the more you give it access to, the better. However, to be safe, the OS itself needs to be locked down in such a way that personal data cannot leave your device. This is why capabilities-based security is an interesting direction: software should not have access to more than what it needs to operate, and you need fine grained control over that.
If you have more ideas, please write us!
[0]: https://en.wikipedia.org/wiki/Software_transactional_memory