> All this infrastructure must be extremely reliable, as we estimate some experiments could run for weeks and require thousands of GPUs

Is it hardward-fault tolerant? Curious how well this will work otherwise as it scales.