Yes - understand first, then fix. And you’ll understand by measuring/profiling things.

I’d also recommend the detailed pytorch optimization case studies by Paul Bridger:

https://paulbridger.com/