Well, it's true that the top and sides of the tube don't get too close to the capsule, so those parts seem relatively low-tolerance, as you say.
But the load-bearing skis ride on an air bearing of 0.5mm to 1.3mm (see page 20), moving at over 1000km/h. As rossjudson notes, the skis are on mechanical suspension, to smooth out shocks to the riders, but it's not clear (to ignorant me) how much of a bump those skis can glide over. 0.1mm, no problem. What about 1.0mm?
On a related note, Musk seems sanguine about the sag you'd get in any structure supported by pylons (see e.g. page 27). Even with inch-thick steel walls, with pylons an average of 30m apart (100'), you'll see some sag, right? Any engineers want to comment on the deviation in 30m of inch-diameter-wall steel tubing? Let's see, 1200kph, 30m, so you pass a pylon 11 times per second. So in 0.09 seconds, you have to go from the top of one pylon, to the valley between two pylons, and back to the top of the next pylon. I guess that's all absorbed by the mechanical suspension?