I think you may be glossing over a bit how annoying / messy / complicated it can be to work with the binary blobs and their quirks, especially when dealing with complex things like camera(s) and the image processing stack. Doubly so for community developers who may have limited or no access to documentation and resources that the device manufacturers get from the more low-level hardware vendors.

If Fairphone wanted, I'm relatively sure that they could form a cooperation with Jolla, Librem, KDE or other entities in the space and get Plasma Mobile, Sailfish or whatever working on their phones and/or design the next-gen Fairphone with hardware that has as few quirks as possible and is as "easy" to get to work with Linux as possible. I'm also relatively sure that so far, that would not have made economic sense for them. However, there seems to be a bit of a surge in people wanting alternatives. The recent Jolla phone has way more orders and pre-orders than were anticipated, for example. And SailfishOS is certainly "polished" enough that the circle of people who can use it well (or at least live with it) has expanded beyond the developer / hacker / activist crowd.