I _wanted_ to drop the plugin and have Prettier automatically run on as a pre-commit hook (as you suggested) but in the end I lost :) We wound up keeping the plugin and eating the slower pipeline time.
Of course there’s more overhead than just Prettier in this case—the ESLint plugin itself will be contributing towards the added runtime!—but depending on your codebase and CI machines, Prettier can definitely slow things down quite a bit. CI taking over a minute or two is a big drag on developer productivity imo.
Ideal world is that everyone is on board with code formatting as a pre-commit task that only touches modified files so it doesn’t bloat CI unnecessarily, but it’s not always possible :(