Is your photo library on a spinning hard drive or USB drive? I had this problem before which the setup, and it went away when I moved my library to my internal SSD.