With any Pigouvian tax[1] you encounter the problem of progressivity. The answer is to make sure that the tax revenues are directed towards making it easier for low income people to have better transit options. Whether that means improved transit systems or income based toll exceptions, it's relatively easy to offset the detrimental impacts that tolling may put in place.