What you call bullshit I call imagination. Both humans and AI need it. Humans use imagination to plan ahead. AlphaGo was generating moves to plan ahead.
Dall-E and GPT-3 are not being used as agents, they are just tool AIs. They have a narrow task - generating images and text. Agents on the other hand need to learn how to act in the environment, while learning to understand the world at the same time.