"which you only have to worry about when you have to tap into the rainy day fund, or when the collateral gets liquidated"
If your 'rainy day fund' is a stack of bills in a box, sure, but other than that there isn't much you can do with significant amounts of money that isn't going to involve some kind of transparency.
Literally trading those dollars for anything else involves gain/loss.
You're not going to be able to 'borrow against' anything under your mattress, it will have to be in some kind of regulated security, a bank account, or real estate - but it's moot - exchanging one asset for another is on the whole financial neutral.
More pragmatically, if you're 'borrowing against sums' you will be paying interest, there's almost a 100% chance you'll want to declare that against any other form of income for the tax shield.
Again, all of this 'hide your money' stuff is really the domain of tax evaders and likely the #1 flag for tax audits.