Not sure where this is stored on Mac, but on Windows, clear the fontconfig directory in the AppData folder.
I avoid it by being careful to not exit GIMP entirely (simply closing all the editing windows) until I am sure I won't need it again for awhile.
A fairly current one is shipped with the 2.9.6 installer, so many of the infamous issues should be gone.
Adding a few new fonts for a specific project does not cause long delays on any platform.
A compounding issue was that file locking issues prevented successful cache updates on the Windows platforms until recent versions of fontconfig, this lead to a long startup time on every start.