That's why it's a land
value tax, on the
unimproved value of the land. Not all land is created equal. Land that's out in the boonies, which nobody wants, is going to have a low value and hence low tax. Land in a downtown area, close to all the amenities, transit, parks, etc. - will have a high value and high tax.
To answer all your questions:
"If I buy a piece of the said land near a town and don't use it for anything I just pay the tax on it and effectively lose money, right?"
Yup. Don't do that.
"What if the city builds infra near my land that makes it effective now to develop? Should I pay more tax now?"
Yes, you should.
This is actually one of the biggest arguments for a LVT, and it has three angles. The first is moral - it's society that's building the schools/parks/transit/infra that made the land value go up, so it should be society that reaps the benefits. After all, the property owner didn't pay to build that subway stop, they just got lucky and got to free-ride off of public benefits. It makes sense that the returns for building public infrastructure go back to the public in the form of additional tax revenue, which they can use to fund maintenance and new infrastructure.
The second is the economic flip-side of this: an LVT provides an incentive to right-size the government. Without this economic feedback loop, there's no check on public-sector spending tax revenues in inefficient ways. Who cares whether a new transit stop benefits residents vs. provides a boondoggle to contractors, if the benefits to residents are all captured by landlords rather than either the residents or the government? But if further public funding is contingent upon raising rents and hence tax revenues within the city, the incentive is to spend money in ways that will actually make the city more desirable and justify those higher rents.
The third economic argument is on individual landlords and developers. An LVT discourages speculation - if you're sitting on that unimproved lot and not getting any revenue from it, you're still paying taxes and will eventually go broke. Your incentive is to sell it to a developer who will build more housing on it, so that more people can enjoy the public amenities that made the land valuable in the first place. Leading into your next question:
"Should I pay more tax in case of a rezoning if I could potentially build a house but I don't?"
Yes. That's how you ensure that land is fully utilized and people who need housing can get it.
"If I can buy and afford (ie have the capital) to hold land for the long term + I can see where new developments are going to take I will be incentivized to hoard land."
It's the opposite way around - you're disincentivized from hoarding land, because you have to pay taxes on the land that's not cash-flowing. You're incentivized to sell to the people who are actually going to build and buy those new developments.
"I can also make the case that it's undeveloped and I should pay less tax."
You can also make the case that the government should just give you money, but in both that and your example, you'd be wrong. Everybody pays the same land tax, based on the unimproved value of the land. A developer who built an 80-unit apartment complex on 2 acres might pay $200K/year; a speculator who sits on 2 acres of undeveloped land also pays $200K/year. The difference is that the developer might be pulling in $2M/year and making a profit of $1.8M/year, while the speculator is slowly going broke. That and the developer is providing housing for 80 families while the speculator is doing nothing useful.