>Users pay the fees both directly (annual costs, interest)

Both avoidable. Many zero fee cards, interest only on monthly balances. Some zero fee cards even offer rewards (read: net positive over cash).

>indirectly (increased prices due to vendor fees)

I'd need to see some data that the convenience of credit cards to both consumer and retailer has increased prices. I do know that explicitly raising prices to cover fees for card using customers is against most merchant account terms of service and you can report retailers that do so.