Because it employs more people and pays taxes locally. It's perfectly valid for the government to ignore that and focus solely on the consumer side (i.e. minimize prices), but it's also perfectly valid (and economically sound) to do what France is doing.
>What is the implied incentive in this case? It seems like it's just incentivizing a brick and mortar business model
No, it is not incentivizing it. It levels the playing field a bit by applying more consistent taxation.
>Doesn't this all point to a fundamental problem with "corporate tax"? Wouldn't it be better (fairer) to reduce or remove corporate tax and raise sales tax or similar, such that it doesn't matter whether the business has a presence in the country in question?
I can be persuaded by that argument (economists do love consumption taxes for a reason, after all), but either way it's not for Amazon to decide. In France, it is for the French people to decide, via their representatives. Whatever positive or negative consequences arise are also theirs to bear.