What kind of GPU are you running this on? My 3080 seems to take about 30 seconds per image with 50 passes. I'm wondering if I'm missing out on some optimizations. Could just be the quality of Linux NVidia drivers.
Are you using 512x512 images or larger ones?
Best workflow is to keep images close to 512x512, record the seed and then upscale.
If you are not using this library already, give it a shot.
Also, I'm using Nvidia Studio drivers though I'm not sure if that would make a difference.