This matches my own experience well: most of my friends do not care about various levels of compression, nor what headphones they use - that's fine, I'm glad they're enjoying art in their own way - but I, and some others, do in fact stand to benefit from less compressed audio.
I've personally done blind tests on myself using a python script that randomly plays compressed and uncompressed snippets of the same track and mp3@320 was not transparent to me (though opus@256 was).
Can I tell the difference when casually listening? I don't know, but when the cost of lossless is having my music collection take 60gb instead of 20gb on my 512+gb device, I have no reason not to go for lossless.