Secondly, I believe that aid to the poor is most effective and of greater benefit to both the giver and the recipient when it is done in a relational context (which is not to say that aid is given directly from the giver to the recipient, just that it's localized enough that the givers and recipients are in the same community). The giver can see the tangible effects of his generosity and the recipient can see the care shown by the giver. By its nature, government aid is impersonal and therefore less beneficial. The "givers" in that case, who are "giving" often only under compulsion, often feel exploited and resentful. Many recipients develop a sense of entitlement since there's no direct connection between the aid they receive and their fellow citizens who provided that aid.
I do agree with you that taking government out of the aid business would result in less uniform funding for aid, but I think that's a lesser problem than forcing it on everyone. I'm on the board of a local charity and I see the private donors who fund it, 95% of whom are are middle or upper-middle class, and I'm pretty amazed by their generosity. I wish awareness of those kinds of things were greater.