Banks exist to make our life better when we need extra spending cash. If we ban lending to the poor than it will be one more thing that only the rich will have.
I agree that the lending rates are insanely high, but that's largely because of defaults. The rates for secured goods (car, house) are insanely low, even when the defaults are taken into account. Car loans from credit unions are below 2% because they're secured.
The real problem is that the easiest way to cure defaults is to start saying "No" more often. But no one likes to be told "no" and that's unpopular too.