It's more effort than I would like. I should not be bothered with updating hidden config files containing data that is mostly not human readable. I want to prove my identity once and let the system take care of the rest.
To me, setting up a CA is more effort than I would like.
I agree with you, for a regular user with a single client device (or two) its not worth it.
The biggest "you're doing it wrong" I see is people who disable host key verification because their servers' IPs change constantly. Do you want MITM?! Because this is how you get MITM! Might as well use Telnet for connections.