Could be a "innocent" mistake. Analysis shows that customers in cluster A are willing to pay 20% more than customers in cluster B for a certain route, as determined by machine learning. Turns out "A" means probably female customer, at night...

I wonder if there are any safeguards in place against this. E.g. take samples of male/female, different ethnicities, etc. that you want to treat identically, and check if they pay the same on average... or put that somehow in the pricing algorithm as a constraint.

I'm sure a naive algorithm with a lot of inputs would, after optimization and without supervision, make single women in bad parts of the city at night pay more. It will probably find a lot more cases to take advantage of, like white (black) people in a neighbourhood where they are the minority and don't feel safe or comfortable. Or rides from places with high percieved crime rates (it wouldn't know the crime rates, it would just know people pay more for certain routes).