Put it this way. Everyone is somewhere on the autism spectrum. I don't think we're over diagnosing it as such, but more diagnosing it at a lower level.
Just because it's a spectrum doesn't mean it's all-encompassing. At some point, it's clear that someone is not on the spectrum. I am willing to bet you're wrong, most of the autism cases are not fringe.