Isn’t this just basic reinforcement learning? We’re not too many steps away from having an AI equipped with a good language model and a reinforcement learning mechanism to be let loose on the internet.
Biological agents have are seemingly able to derive goals from both internal and external signals, and are able to use affordances (aka what is usable through their environment/body) to accomplish those goals. These affordances are not predefined and the space of possible usages for any given tool is practically infinite (a screwdriver can tighten a screw, or pry open a door [0]). Biological evolution also doesn't actually have goal persay, its more of "what works sticks". Whereas evolutionary/learning algorithms have a defined objective function.
The big differentiator seems to lie in the fact that biological agents are able to engage in a semiotic or meaning-making process where given an internal state and its affordances the agent is able to make movements that betters it's state in the future without explicitly being told.
[0] https://www.frontiersin.org/articles/10.3389/fevo.2021.80628... (title: How Organisms Come to Know the World: Fundamental Limits on Artificial General Intelligence)
I wonder if Pearl thinks that RL replicates his do-calculus under the hood, or if that's an innovation we're missing.