What the masons in the video do is certainly impressive. Cutting organic shapes that fit perfectly together, as if they once were elastic, is another level.
Perhaps the did something similar to what dentists do when building on teeth so that the added material is not the only contract point when jaws are closed. That is, a contact sheet that leaves contact marks.
I know. I mainly just wanted to link that video because it is awesome.
The article does explain how the Inca did it - only the front edges are tight fitting. The gaps between the inside surfaces are filled with mortar. They sat the stone where it was to be placed, but with the front edge raised up by resting on some spacers, then just incrementally improved the fit of the edge and re-tried the fit. I'd have still thought that was impossible without seeing something like the video I linked - my intuition of what can be achieved with hammer and chisel was wrong.
The article linked in this post mentions the possibility of „red clay“ being used for this purpose, as well as being a mortar.