Gotta be taller than the tallest ladder would be my guess. Tower forts are common. You can find them in places like Ireland also.
This is why they build seige towers back in the day - to give the attackers a position higher than the beseiged city's walls from which to fire down on the defenders.