I have read differing definitions of "AGI" but we can start with the Wikipedia definition unless you disagree:
> Artificial general intelligence (AGI)—sometimes called human‑level intelligence AI—is a type of artificial intelligence that would match or surpass human capabilities across virtually all cognitive tasks
We don't have consistent metrics for what we might mean by "human-level intelligence" or "cognitive tasks." We have some tests and benchmarks, with obvious flaws. For example LLMs can already perform well on SAT tests and legal bar exams because those get included in their training data. A category error, in other words.
I'll give a real-world example from my experience. My wife had to pass a US citizenship test, described by USCIS: "The USCIS officer will ask you up to 10 questions from the list of 100 civics test questions. You must answer 6 questions correctly to pass the civics test." USCIS and others helpfully publishes the 100 questions and answers, so like so many candidates for US citizenship my wife memorized all of them (and passed). She mimicked an LLM. Of course she has a very different understanding of US history and civics than I do, because I grew up in America and attended and got socially programmed in public schools. My wife can perform better than a typical native-born American on that test, but has little to no real understanding of the material. Optimizing for the test -- Goodhart's Law illustrated -- seems more like what LLMs do, and it looks like cognitive ability because it succeeds at the metrics.
In my last comment I claimed you make a category error by applying words such as intelligence, mirroring, bonding, evolution, objectives to LLMs (or AI if you prefer). While those words have vague or flexible meanings when applied to animals or humans, they don't apply at all to computer hardware or software. We use words like "intelligence" and "objectives" when talking about computer programs as analogies and conveniences, and then quickly fool ourselves into equivalency.
The concept of AI "alignment," derived from the paper clip maximizer and seen in 2001 and The Terminator movies, seems hard to take seriously. It comes from sci-fi and paranoia about humans making machines they can't control (a trope as old as the Industrial Revolution). In many ways we already live in a world where machines and software operate in ways not aligned with human needs or objectives. When my iPhone once again fails to pair with my Airpods I could say my technology has not aligned with my purposes. What objective does my phone have when it acts up? What evolution or self-editing has taken place that makes it work one day but not the next?
I can align my phone by turning it off and back on. If I perceive that the machines I use or the software I interact with no longer align with my goals I can just switch it off or delete it. Barring sci-fi scenarios where the machines break into the physical world to protect themselves from humans turning them off I don't think we have to worry about it. Humans have lived with dogs for millennia, and we know what happens when a dog's behavior fails to "align with" the humans it lives with. We treat our machines and software the same way with far less empathy.
The stories about LLMs appearing to have their own objectives, lying, protecting themselves from shutdowns, etc. amount to hype from self-interested hucksters like Sam Altman. The AI marketing narrative -- which exists solely to keep the money flowing in -- generates scary stories to give the appearance that AGI lurks just around the corner, about to "emerge" from millions of GPUs. Look into the contract OpenAI has with Microsoft and it seems obvious why Altman pushes the imminent AGI story.
We could speculate on the dangers of humans evolving the ability to read minds, or see into the future (both explored exhaustively in literature and movies), and then wring our hands over what to do if real X-Men somehow evolved. To me the (mostly fake and cynically manufactured) worry over AGI falls into that category. I could have it all wrong, we will see. If some Silicon Valley startup got hundreds of millions to develop psychic humans, or cognitively enhanced humans (see: Neuralink), would we interpret their claims and ethical quandaries as real or just cash grabs? The AGI threat works because the general public doesn't understand how it works and has seen both benevolent and malevolent AI in movies for decades. Most people can't explain how a microwave oven works despite living with them for decades. I date back far enough to remember when my grandparents feared a microwave oven giving them radiation sickness, at a time of widespread public ignorance and worry over anything "nuclear" and the word "radiation" jumping from science into everyday discourse (1960s).
We have had software that can write other software, and edit itself, since the 1950s. I wouldn't use the term "evolve" to describe what computer hardware and software do, but we have seen amazing progress (through human insight and effort driven by profit, not Darwinian evolution) along multiple axes such as performance, size, power consumption, and price. Then physical limits get reached and the technology plateaus. I expect LLMs will plateau in the same way, due to inherent limits of the approach, exhausting the training data, or maybe power/water requirements. At this point LLMs amaze us in a way other software does not, because of natural language mastery. I think we have mistaken that impressive but narrow achievement for overall cognitive ability, because we only have our own species to measure against, and among humans mastering language and appearing to "know" a lot of things get interpreted (rightly) as intelligence.
AI and what some will surely eventually label as AGI may pose an actual existential threat, the way nuclear weapons and tinkering with viruses already do. I think more likely the AI companies pose a grave economic threat, because if the bubble bursts before real use cases and profits materialize we will all suffer. The dot-com bust, 9/11, the 2008 real estate collapse, and COVID give some idea of the economic and social disruption (and government overreach) that happen even without Skynet trying to kill us due to mis-alignment.
P.S. I have seen links for LLMs doing horoscopes, tarot readings, palmistry, fortune telling, and Bible study promoted here on HN. That says more about human ignorance and credulity than it does about LLMs, but as long as the software happily goes along with the nonsense and doesn't scold the vibe coders for asking I will sleep easier knowing the LLMs have remained in alignment, and haven't given any real deep thought to what we ask them to do -- at least no more thought than a backhoe gives to its work, or my phone gives to pairing over Bluetooth.