This is my biggest concern as well. Just to address your points:
* My firm belief is that once felons do their time, their debt is paid and they should be full citizens again. They should definitely have the right to vote, although gun ownership might need to be addressed on a case by case basis for violent criminals for the safety of society. So UBI might need nondiscrimination laws attached to it.
* If UBI is per capita, then a single mother with 2 children who doesn't receive child support would immediately see a roughly 3x increase in funds. This could be used to dissuade special cases for allocations.
* Rent will almost certainly go up in places that have no appreciable disposable income currently. This may be a straw man argument however, encapsulating the fallacy that inflation would increase with UBI. If no additional money is being printed, then prices will even out somewhere else. For example, people might be able (as a whole) to afford mass transit or urban renewal projects which would open up competition for cheaper housing in other areas.
Aside from specifics, I see your point however that people working for the general welfare may inadvertently undermine the efficacy of UBI. This is probably the single biggest threat to it, since I don't see any serious technical or financial hurdles.
I'd vote to start everyone with a UBI plus a stipend that adds up to each welfare recipient's current payment (financed with debt initially). Then put a hold on welfare increases for 3-5 years. My guess is that the federal government wouldn't end up paying substantially more than it does now, although I'd like to see a study/projection on that.
Also this all needs to happen outside of self-funded agencies like social security. I think that sacrificing so-called entitlements for UBI would be unethical, since people have paid into social security their entire lives.