If it was your one friend who got drunk and in a bar fight and charged with assault when he was 20, maybe you'd think differently :)
Even if you restrict your view to felons, in california, for example, there are plenty of assault charges that are easily classifiable into "stupid thing you did once and did not cause long-term harm" but still felonies.
At some point, what folks are really saying is "i'm okay assuming this person is a bad person likely to do something bad". This is, for better or for worse, at this point, pretty unrelated to whether they have a criminal record or not.