The biggest issue is probably just trying to find a return fair. Late at night most people are going from Manhattan to one of the outer boroughs - not the other way around. The usually end up driving an empty cab back to Manhattan, so they are effectively getting a half-fare.