The "not too loose, not too tight" thing is part of it, but isn't actually the key reason which is "accumulation of tolerance". Imagine two rows of 100 bricks. The difference in length between the two rows is 100 times that 1um tolerance. If the tolerance was less good, things wouldn't fit together well.
Lego model design guidelines actually have a maximum size of a "fully" connected block of bricks because eventually even this precise manufacturing can't cope. Really big models have purposefully designed decoupling between sections.