How large are we talking here?
I'm currently sitting at 35k~ tracks, primarily FLAC with some 320kbps mp3 sprinkled in. I also use Jellyfin for anime and have a LOT of shows on there. Haven't noticed any issues yet, so I'm curious where your tipping point was.