The same fear will happen if you just target investors owning multiple real estate with this tax, or simply forbid by law from owning several flats in high demand areas. The right wing would scream that the hard working French guy won't be able to invest his hard won money, but the very rich foreigners from Saudi Arabia or investments funds from USA will find a loophole thanks to their infinite money and buy all the french real estate.
Hell, most people in my country are against inheritance tax despite a huge part of them not rich enough to pay it, meanwhile inequalities are rising because of inherited wealth. So taxing the land won't happen, the bourgeoisie has been too effective in its propaganda.