While I agree with your last sentence, I believe you are misunderstanding the purpose of the score.
It is meant to facilitate your risk assesment. Taking your example, A lot of people think RCE is automatically bad and XSS not so much(but yeah,self-XSS is mostly low sev). If the vuln is regarding a bug where theoretically RCE might be possible but there are no known exploits then it should have a lower exploitability score than unauthenticated XSS on a login page right? Similarly, naive people might think DoS isn't so bad but service disruption can be a lot worse than RCE on the asset,data and threat model. CVSS is meant to guide those people so they prioritize based on a standard score rather than intuition.
Threat model is another thing,if you expect attackers that can rapidly develop exploits after discovery then you would increase/adjust the exploitability score yourself (and the score does change over time as exploits are made public).
I guess my perspective is that hygeine is hygeine,either you have it or not.all vulns should be remediated regardless if any scores. I have been responsible to give guidance to IT ops teams on prioritizing patches and this helps,but you have to understand the purpose and meaning behind the scores. It can certainly be misunderstood and I am sure there are many improvement opportunities like what you mentioned but the alternative is to expect resource limited staff to read and understand every vuln (including how specific complex attacks can be pulled off). And even if that was possible,how does someone manage said staff validate things are being prioritized and properly? How do you get an external auditor to properly assess your practices if you interpret every vuln autonomously without any well established and understood priority structure. I mean you can change the score according to your opinion all you want but at least you can say "Because of _____ I am increasing the exploitability score" as opposed to "I dont think it's so bad so I'll score it n". Maybe I just have a different perspective.
Haven't heard of OSVDB in a long time but it exists too. Most orgs need something to feel this need.