That's not really likely to help. The main issue appears to be that since fontconfig (a lib used by GIMP) is only "native" to Linux, for it to know about the TTF files, it has to scan them all and build it's cache at startup. For people with hundreds of fonts installed it can be a real PITA.
I avoid it by being careful to not exit GIMP entirely (simply closing all the editing windows) until I am sure I won't need it again for awhile.