Every validation you do on a mock is an assertion on outside behavior that is in no way coupled with your test. If that behavior changes, your test is now false. Not failing. False. That concerns me.
I can imagine scenarios where that is useful and I know how to use mocks when that comes up ... but the examples are always a system where mock based testing is strangely coupled to implementation, very fragile and have the scary failure mode you describe.
Side effects can be things like making an http request or writing to a file, it can be changing some global variable, but it can also be something like calling a public method of another public object. Since that function/method could also be impure, the very act of calling it is actually, in a way, part of the contract of the function under test: it returns a value, and it creates a side effect in the form of calling another function.
The only real way to test for those side effects is to pass in a stub or mock and to check that the function was called correctly. That makes your tests brittle, but it's the only way to make sure it does what you think it does (even if you only call the function for an intermediate return value, the fact that changing the intermediate function could change the result of the function under test).
Those tests are brittle because functions with side effects are, in fact, brittle, and favoring pure functions (where the only thing that matter is the return value, and dependencies are composed or passed as arguments) makes both your tests and your code better.
This way can lead to brittle tests, clearly. But the reality seems to be that brittle systems lead to brittle tests. There is no testing panacea.
Mocks lead you down a path where developers will eventually feel too scared or fatigued to refactor anything substantial.
then you have technical debt to repay, by making those pre-conditions satisfiable