I thought the same until I watched it a second time.
4th dimension is still time. 'The button' takes you to your character's view of moving across a 3rd dimensional plane. I'm assuming the two different textures on the ground represent the change in the 4th dimension (time).
So when you cross to that other side and hit the button again, you're back in the 3rd dimension across a different plane (time). So the desert area is either in the past or in the future (when the wall fell or was being built).
Then he goes through the wall and hits the button again and walks back to his original 3rd dimension.
That's what I took from it anyway. Seems really neat.