The problem with for-profit is that it informs priorities and incentives, usually not in the direction of societal benefit. That’s why the phrase “selling one’s soul” is being applied.
It's really the bedrock of modern economics. Invisible hand of the market and all that.
The entity that first achieves real general AI has the potential to achieve a decisive strategic advantage that will allow it to shape the world in its image, free of competition. Traditional economics might not really apply anymore - the values that are built into the constructed intelligence will be paramount.
It won’t just all work out in the wash if we end up creating a superintelligence with values that are not to the benefit of humanity at-large.
The focus on "general AI" is so so so shortsighted. You don't need general AI. Really, you don't want it. It's far better to have tools that you can control and direct that are specialized in specific areas. Specialized AI tooling is a force multiplier that can be leveraged however you want, general AI is an employee that might disagree with your goals.
I'm not scared of a superintelligence, I'm scared of AI being O'Brien's boot stomping on the face of humanity forever, in service to the corporations that invent it first.