Maybe make it a two-step process to add people to a group. Sure, show the list of friends to suggest, but make the "adding friends" page separate, so if you accidentally click, you're still a step away from adding someone.
Or better allow people to mark groups as more private (lets call it sensitive) and only if done so have a two-step process to add people and possible other changes.
Product people are going to hate that friction, just add an undo invite button
That would work until another product person decides to eliminate the friction of "accept invitation" and adds an auto-accept invitation feature