Since the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.