I don't understand how "safe" AI can raise that much money. If anything, they will have to spend double the time on red-teaming before releasing anything commercially. "Unsafe" AI seems much more profitable.
Related to this, DAO's (decentralized autonomous organizations which do not have human shareholders) are intrinsically dangerous, because they can benefit their fiduciary duty even if it involves causing all humans to die. E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.
https://www.businessroundtable.org/business-roundtable-redef...
The idea behind "corporations should only focus on returns to shareholders" is that if you let them do anything else, CEOs will just set whatever targets they want, and it makes it harder to judge if they're doing the right thing or if they're even good at it. It's basically reducing corporate power in that sense.
> E.g., if the machine faction in The Matrix were to exist within the framework of US laws, it would probably be a DAO.
That'd have to be a corporation with a human lawyer as the owner or something. No such legal concept as a DAO that I'm aware of.
We can’t build critical software without huge security holes and bugs (see crowdstrike) but we think we will be able to contain something smarter than us? It would only take one vulnerability.
enterprises, corps, banks, governments will want to buy "safe" AI, to push liability for mistakes on someone who proclaimed them "safe".