I had constant syncing problems between Infuse and Jellyfin previously. The first sync after clearing the cache would succeed, but all subsequent updates would fail with a nondescript error on the server. I managed to fix it by scanning for errors in media files by writing a script to walk the library and output ffmpeg copy errors, and either repairing or ditching the broken files. I guess it was just choking on one of them.