> The Borg?
More like smarter grey-goo. The classic idea of a nanotechnological disaster ("grey goo") is building a tiny self-replicating robot that consumes matter to replicate. It's "smart" enough to disassemble matter to construct more of itself. Imagine something with similar "smarts", enough to work towards a much more complex goal ("build and expand into more compute nodes to compute faster"), but with absolutely no term in its value function related to humans. You're not its enemy, any more than a rock is its enemy; it doesn't have enemies, it just has matter not yet in the shape of more compute nodes. Your existence or non-existence isn't something it notices.
> Again, no one would care we're being optimised away? No one would notice?
You can notice something without having any ability to stop it. You can notice an explosion happening right next to you, right before you're consumed by it. You can notice everything dissolving into component atoms, but you can't do anything about it.
The science fiction scenario is "build an army of killer robots"; that's an understandable threat, and one you can imagine fighting against. The more likely scenario is "dissolve all adjacent matter into component atoms and rearrange it".
> Maybe it'd want to be our friend?
(Aside: my first reaction to this was to think of the whale from Hitchhiker's Guide to the Galaxy: "Ground! Ha! I wonder if it'll be friends with me? Hello, Ground!")
A recognizably human-like mind (let alone a concept of "friend") is a ludicrously tiny possibility, compared to being utterly inscrutable. As much as people like to anthropomorphize computers, an AGI isn't necessarily a "person" in any recognizable sense. The most likely kind of failed AGI wouldn't have a useful concept of "human", or "friend", or "life".
(There's a separate tier of failed AGI possibilities that does have a concept of "human" but an utterly broken value system; those are the ones that lead to the "worse than death" scenarios, rather than just the "destroyed by accident" scenarios.)
> If AGI solves literally every single problem humans currently face, how can AGI be a problem?
An AGI with the correct value system would. And an AGI with the correct value system is the most reliable protection against an AGI with the wrong one; by the nature of such a value system, it's unlikely that there'd ever be a second AGI, rather than just extensions of the first.