The disabled are a concentrated special interest group, and we’d always want to keep it that way. But who will nerd out for the disabled if not for those among them who seek disproportionate impact relative to their base numbers?
The disabled are in fact a concentrated interest group!
I think you might be letting a non-conventional communication/thinking style cloud your ability to communicate in the political space. If you say the we should ignore concentrated/special interest groups (which is normal - colloquially the term refers to nefarious groups with disproportionate power), but also that disabled people are a concentrated/special interest group (which is literally true, but not aligned with the colloquial definition), then you are saying we should ignore disabled people, which is not the same thing as treat with proportional weight. But I think you're not literally trying to say that.
No, I think disabled people are very important and their needs are important. I just don't think they should get an extra vote.