That's a common point of view, but when your disability is never someone else's problem, it becomes waaaaaay harder to manage. You should display more empathy to people that don't follow the norm.
I try to be generous as much as is reasonable. I generally assume the person who cuts me off in traffic may have an urgent need, but vaulting every misdeed to an assumption that it's due to some unknown disability crosses into unreasonable territory, if for nothing else than it's probabilistically a bad assumption. Taken to the extreme, it becomes enabling for everyone who does not have a disability but gets away with bad behavior.