Non-potable is the industry term for "non-human drinkable".
And yes, the cost of that would be huge, not just in cities. In the UK the decision the victorians made to not have separate sewage and rain water drainage systems still has a knock-on today and costs a vast amount in water treatment, but the cost of separating rain runoff and sewage is still seen as prohibitive.
edit:
However, on a small local-scale grey water systems can do a lot of good, both for re-use but also rain water capture and re-use would help with flood prevention. If houses captured rainwater for use for lawn watering and other appropriate uses, this is something that could be done without great expense.