What "industrious poor" people are you thinking of who own extremely valuable land but don't do extremely valuable things upon that land?
Most of these people are middle class, many are cash-poor.
Property developers are a stark minority, you know...
...And, again, without recourse. Nobody has to gamble or buy a yacht. But whether renter or owner, everybody needs a house to live in. Everybody would have to pay the LVT, one way or another, no matter how rich or how poor.
For the average landowner, the LVT is actually a sort of tax on sober industriousness or fortunate inheritance, the very opposite of a vice tax.