You realize those who think it's important don't think that's what it is, right?
If 20 employees were on the "social" committee planning a holiday party, would you consider that as "off the rails", because you are paying them (some here unspecified number of hours) to throw a damn party? I doubt anyone would get that up in arms about it.
I get it, you think DE&I is a mistake, (unless you think coming up with new ways to tell people to stop being racist is not a mistake? But I think you were clear), so the more people involved in it, the bigger the mistake. But if someone thinks it's important, perhaps even more important than a holiday party, then the more people involved in it the better, and having a large portion of the company involved in it is not a mistake. That's it.