If it were based on open ended tooling that anyone interested could experiment freely with, I would feel the same. Unfortunately these giant LLMs are almost the complete opposite of that.
I would love to work on doing something like that.
Furthermore, how would upgrades work? When new techniques are discovered, it would be hard to upgrade it. We see these challenges already when it comes to major changes in cryptocurrency networks.
Somehow, the AI believes that Alan Shepard was the first person on space.
Doing this session reminded me A LOT of Asimov's short stories in "I Robot". I feel that the future AI "debugging" session will be like that. It's been a while since I read the book, so I'll have to re-read it again.