The trick here is never turning it off so the ICL keeps growing and learning to the point where it’s aware.
We will have something we call AGI in my lifetime. I’m 42. Whether it’s sentient enough to know what’s best for us or that we are a danger is another story. However I do think we will have robots that have memory capable of remapping to weights to learn and keep learning, modifying underlying model tensors as it does using some sort of repl.