To your basic point, fully agree. I expect that at least some of the carryon issues are the result of business travelers taking rollaboards for one-night trips. Like you, I find soft over the shoulder/backpack bags a lot more versatile as well as being more space efficient. Yes, some people can't easily carry the weight but that describes a small portion of travelers.
I do have rollers on a couple larger pieces of checked luggage but I only use those a couple times a year.
Which one? This sounds like good advice, but I just did a quick search [0], and in the first several results saw only hard-sided spinners and a backpack.
[0] https://www.amazon.com/s?field-keywords=Amazon+Basics+carry+...
The wheels on two-wheeled models don't intrude very much, I would definitely avoid the 4-wheeled models, you lose so much space to those caster wheels.
I thought about getting a backpack, but my back gets sweaty within a few minutes of walking with it.