By that logic it's also regressive to give rich people free access to roads, free access to public K-12 education, etc. Money is fungible, and if progressivity in an income tax pays for the "free" resource, it's disingenuous to call it regressive. It becomes a matter of practicality. Do you want both an IRS and a Department of Basic Income Administration?
Also, by definition a flat tax is neither progressive nor regressive. , so I'm pretty confused about what you're talking about. Note, flat doesn't mean fair by everyone's definition.