Cashless society in Sweden is very much built around a Swish, a local service that I don't think has much to do with Visa/Mastercard as it communicates directly with your bank.
The bank still charges 2 SEK / transaction (~$0.2). This is a free handout to the banks, which already profit from keeping your money in their books. It's also a fee that affects mostly small vendors.
why don't vendors forward that price on customers who decide to use that payment method?
What do you think handling cash cost?
how is that relevant? Why don't vendors adapt the final price to the payment method?