I agree entirely, mostly because there's not a huge difference between that and simply not carrying a phone... or your battery is dead unless you carry a bag full of spares.
The state of non-Android Linux phones (PinePhone, Librem 5, etc) leaves an awful lot to be desired, and I'll argue that for most of the stuff that actually works on those devices right now, you're better off carrying a random flip phone that works on modern networks, with week+ battery life. They'll do about the same things, but the flip phone is far cheaper and more likely to actually send/receive phone calls if you need it to.
I'm glad the non-Android Linux devices exist, I have a PinePhone I'm messing with, but they're not really daily driver grade yet, without quite a bit of pain.
The problem with this is that a lot of services don't make apps for this (banks, gov't services, messenger services, etc.) and that the value of such a phone is hugely reduced.
It's basically Microsoft Windows anno ~2000 all over again, except now we can "choose" between Android and iOS (marginally better, I suppose).