I don't exactly agree that it's a design problem. I think there is a lot of value in testing at a high level (controller tests) and having them touch the full stack (down to the DB). Gives a lot of confidence that the code will work in production. Throwing money at it (running the tests in parallel) pretty much solves it.
I have seen FactoryBot use get out of control (creating 8x as many records as you'd expect). That's a really easy way to slow down a test suite :). One way I've found to fix that is by adding tests for the factories, and asserting they are only creating what you want them to.
On model tests, another thing GitHub did well was encourage using `.build` when possible to avoid writing the the DB.