I wonder if it was just a start up issue? Some kind of initial scan of your database?
Once on the client side, it felt like those also needed to cache things. The only client that worked semi-well wasn’t a Jellyfin client, but could read from a Jellyfin server. I think it was called InFuse. With that as well, I spent hours simply navigating the app, going to every page, and waiting for it to cache everything so I could then navigate around. Then if I added something new, I needed to go to a page to show it and get it cached.