Your mortgage is itself an asset. It’s one of the best hedges against inflation - as long as you don’t fall into the refi trap.
Imagine you have 20 million. Should you buy a million dollar house outright, or get a mortgage? Look at the prime rate today and compare it with expected market returns over the life of a mortgage.
This is without even considering the ground reality of renting a place (which is the common alternative), where rents are liable to rise perpetually, and without considering a plethora of other factors which contribute to calling a mortgage an asset.
Of course, if you can get by without needing to pay for shelter at all, you should do that.
The mortgage is the opposite of an asset.
Personally I argue to everyone they should put down as little as possible for the reasons you're stating.