> I am in no way poor, but if I lost my job and couldn't pay my bills, I'd be in trouble very quickly
Sounds like if you lost your job you could pay the bills. You're running into the difference between what is - for you - a hypothetical and what is for others a reality.
> That's no one's fault but my own
Bu-bu-bu-bullshit! This is completely discounting the role the entire rest of civilization has in producing the context within which you make your choices.
Here's an example: I met a homeless old couple at an encampment once, and they were there because they'd chosen to move out of the Section 8 housing they were in because the building was becoming a drug den. They chose to be homeless but I do not think it is reasonable to say it's "their fault".
I think what you're actually saying is "I don't want any responsibility for other people" which is fine, but realize a) there's a difference between that and someone's situation being their fault, c) a moral judgement by you is inappropriate, and b) "If Everybody Did" then we wouldn't have civilization. Some people need to, do, and should take responsibility for others, and you should disparage neither the needy nor the charitable on those grounds alone.
(Note: If everybody takes all responsibility for everyone else, you get tyranny instead. Balance and diversity!)