Since you don't seem to be arguing, I would like to understand what you mean? Did I use such a pejorative? I'd sincerely like to know - if, that is, your query is sincere as well.
Let me put it this way. If I call someone cancerous, I'm not trivializing the group of people that have cancer and distilling them to a pejorative. In general, if I use a disease as an insult it's fine, but if I use a disability as an insult it's usually offensive. They're both afflictions, but they're treated differently in uncountable ways.
I could list some reasons, but none of them really feel like they get at the root cause of the difference. Do you have any insight?
(I don't have first hand experience with either, so I'm just musing too..)