A brittle test suite during refactoring is usually caused by too much mocking.

Look at the concept of sociable tests.

https://martinfowler.com/bliki/UnitTest.html