Unless the human brain is made of some sacred substance, the worst-case scenario is that we will extrapolate current scanning methods into the future and run the scanned model in silica. I'm not recommending this "just for fun," but the laws of physics don't forbid it.
When we are successfully turning base metals into gold, hit me up.
So is Sam Altman the modern day alchemist? Making predictions based on faulty methods and faulty understanding (per your gold example)?
What will happen is that we'll shift the economy around based on inflated tech promises and ruin people's lives. No big deal I guess.
Alchemists were early scientists who later branched into fields like chemistry, mathematics, and physics (Newton explored alchemy).
Altman leads a team of experts in neural networks, programming, and HW design. While he might be mistaken, dismissing him outright is difficult.