Build with stone blocks. Even if it gets knocked over the blocks just need to be reassembled to be useful whether as a wall, a tower, a road, or another house.
NB I did this because our house is an old Scottish farm building that was converted to a house ~12 years ago - I wanted to be able to do proper wall repairs and build garden walls in the same style.
If you see experienced bricklayers at work, the speed with which they go and the perfection of the result then that's always a good reminder that plenty of the 'trades' that IT people tend to look down on are actually highly skilled professions that can take the better part of a lifetime to master.