I'm only addressing a fraction of your many excellent points, but: this is actually why uncontrolled AGI is so spooky. It's the prospect of a future where space gets colonized-or whatever crazy goal(s)-but there's "nobody home", no qualia, no experiencer.