It can run much faster. For example, using the PyTorch nvFuser JIT gives a 50% speedup:

https://old.reddit.com/r/MachineLearning/comments/xa75km/p_p...