You don't have to pay the driver. Currently it's more expensive for me to take a taxi (in Australia) because the wage of the driver outweighs the extra costs (and inefficiency) associated with running my own car.
You see a lot more taxis in cities with lower human labour costs (such as Lima), or where the cost of parking outweighs the cost of the driver (such as Manhattan).