The concern is more with second order effects. If potential investors who don't have access to insider information think the market is rigged then they'll be less likely to invest, thus increasing the cost of capital for businesses.
Ironically I've heard the exact opposite effect being proposed as a reason for allowing insider trading. If you allow people to profit from their knowledge about inner workings of a company then that information will find its way to the market faster since people are incentivized to take a position and then capitalize on it. Not sure if that's too simple of an explanation but it seems facially plausible. In this framework the market would have more timely information about the state of things so investors could potentially be more likely to invest overall.