Financial management is about much more than maximizing returns. It's also about managing risk. For most people who have assets that they will depend on in the future, managing risk grows in importance as they grow in age. Taking on debt (a mortgage) in order to make speculative investments is a high-risk endeavor.
Implicit in that was that the investment is risk-free i.e. it's a government bond or something. But it could be a risky investment, if the risk-return trade-off is high enough.
If I had the opportunity to invest at a 10% rate of return with a stdev of 5% then absolutely, I'd remortgage my house at 4% to do that!
1) It is difficult to take a house away from you if you are worried about "bail-in" type scenarios.
2) Paying off your mortgage is a guaranteed investment (there is no default risk) and so it should be compared with long term investing in very high quality government bonds when looking at rates of return, rather than what you can achieve more broadly.
3) It eliminates any future call risk, where your broader investments tank and it becomes difficult to meet your fixed mortgage obligation without liquidating investments at fire-sale prices.
4) It is perfectly hedged against future housing costs, which you will probably always need.
5) This is controversial, but if you believe, as I do, that a fractionally reserved monetary system is immoral, paying off your mortgage (and all the rest of your debt) is as hard a blow as you can land against it.
I take issue with (4) because if you have a mortgage you are already perfectly hedged against future housing costs. If you have a floating rate mortgage you are exposed to interest rates, but you can hedge against that by fixing your mortgage.
I think a blanket "debt is bad" approach is naive, just as much as a "risk is bad" approach is naive. All other things being equal, you would rather not have any risk or any debt. But for the right price, you should be willing to take on both. What 'the right price' is will depend on many things, like your stage of life, your income, your safety nets (the state, your family) etc. If you take a "debt is bad" approach then you are saying that the right price is infinity, which doesn't seem sensible.
I agree that "debt is bad" is simplistic, but good debt, in my mind, is self liquidating debt taken on for productive enterprise. Mortgages are not self liquidating, and consumer credit is of course much, much worse. Mortgages at least have the benefit of good rates and tax advantages, although the size of them can often make the practical cash-flow ramifications dicey.
You are right, of course: at some rate of return, a sure thing at 3% becomes less attractive than an alternative investment. And you have to look at your broader investment basket to put together the appropriate mix of risk and returns. If paying off your mortgage involves say 20% of you net worth, that's different than 90% of it. So, as always, it depends.
I do like the thought experiment where you ask yourself "If I had my house paid off, would I take out a mortgage to make this investment?"
You might not be alive later. Don't put everything off until then.