Very location dependant.
In my city the majourity of panhandlers you see have homes, and choose not to work and beg on the street. You can tell who's homeless because they don't bother you, they sit in silence, or in front of a shelter. Or they ask for food rather than money.
People harassing you for money? Panhandlers.
I'm sure it's different in cities that have higher costs of living s.t. a min wage earner can't afford rent anywhere.