I think you might be letting a non-conventional communication/thinking style cloud your ability to communicate in the political space. If you say the we should ignore concentrated/special interest groups (which is normal - colloquially the term refers to nefarious groups with disproportionate power), but also that disabled people are a concentrated/special interest group (which is literally true, but not aligned with the colloquial definition), then you are saying we should ignore disabled people, which is not the same thing as treat with proportional weight. But I think you're not literally trying to say that.