It discriminates against everyone except the minority of public administrators it ensconces. It literally removes the discretion and ownership of money if you cannot physically possess it. It polarizes people involved in grey market transactions into a permanent underclass who cannot escape it, and just partial cashlessness has been used within the last several or so hours to disenfranchise political opposition. That it is being discussed seriously at all is an offensively false equivalence. The only people who are "cashless" in a cashless society are the citizens from whom the cash is taken.
The arguments back like, "I have nothing to hide," or, "cards are just convenient," aren't centerist or neutral, they are the banal nihilism of people suited to scheduling prison trains, imo.