A city is a state actor for Fourteenth Amendment purposes. The Supreme Court has applied the Fourteenth Amendment to municipal action since at least Yick Wo, and municipalities can be sued under §1983 for constitutional violations caused by official policy.

And "you can vote them out or sue in tort" doesn't answer the constitutional question. The availability of other remedies does not make an otherwise unconstitutional government action constitutional.

Also, the Fourteenth Amendment does considerably more than prohibit abridging privileges or immunities. It expressly prohibits states from depriving any person of life, liberty, or property without due process and from denying equal protection.