Right you are, and the land tax is specifically designed to destroy the speculative activity that causes it to increase forever.
How are we determining that the land value increased because of speculation, and not because the land continues to become exponentially more valuable do to its location in a popular area?