I've ran into this decades ago with file servers back. Half a million files, takes too long to do a simple for loop and calling 3rd party processes for awk/sed when I was just using them to format/search text. Breaking it down to just to mosty bash one scripts reduced the run time and ended all pauses.