Oh, but that money you borrowed isn't getting any investment returns in our 401k (I hear people say). But it is -- it is getting a 4% return (what you are paying back in interest). And considering that a well balanced fund is going to have some amount in a lower fixed-interest investment, that isn't much of a problem (just rebalance the fund when you take your loan out, then rebalance again as it gets paid off).
The ONLY downside I see, is that you have to pay it back all at once if you lose your job, or face a 10% penalty (plus tax) on the loan balance.
Some plans do not permit net-new contributions while a loan is outstanding. Even when plans permit it, the fact that you're borrowing on Thursday might not have you in a situation to make payroll deductions on that Friday. (This is situational, of course. If you're in dire straits overall, you might still not be making contributions.)
I'd consider borrowing from a 401k to buy a house, or to pay off credit card debts that were 15% or worse and that's it, full stop. Refinancing 7% debt at 4% seems not worth it, especially if you're taking money out of equities to do it.