—-
Ilya’s success has been predicated on very effectively leveraging more data and more compute and using both more efficiently. But his great insight about DL isn’t a great insight about AGI.
Fundamentally, he doesn’t define AGI correctly, and without a correct definition, his efforts to achieve it will be fruitless.
AGI is not about the degree of intelligence, but about a kind of intelligence. It is possible to have a dumb general intelligence (a dog) and a smart narrow intelligence (GPT).
When Ilya muses about GPT possibly being ephemerally conscious, he reveals a critically wrong assumption: that consciousness emerges from high intelligence and that high intelligence and general intelligence are the same thing. According to this false assumption, there is no difference of kind between general and narrow intelligence, but only a difference of degree between low and high. Moreover, consciousness is merely a mysterious artifact of little consequence beyond theoretical ethics.
AGI is a fundamentally different type of intelligence than anything that currently exists, unrelated and orthogonal to the degree of intelligence. AGI is fundamentally social, consisting of minds modeling minds — their own, and others. This modeling is called consciousness. Artificial phenomenological consciousness is the fundamental prerequisite for artificial (general) intelligence.
Ironically, alignment is only possible if empathy is built into our AGIs, and empathy (like intelligence) only resides in consciousness. I’ll be curious to see if the work Ilya is now doing on alignment leads him to that conclusion. We can’t possibly control something more intelligent than ourselves. But if the intelligence we create is fundamentally situated within a empathetic system (consciousness), then we at least stand a chance of being treated with compassion rather than contempt.