My very biased opinion: you start with PyTorch because it's easy to develop and debug, and there's no point in having the fastest tools for a model that you can't train properly.
Once your model is running, and if/when you start hitting performance bottlenecks, then you consider migrating your model to TensorFlow.