The optimiser doesn't know how long optimisation will take or how much time it will save before starting the work, therefore it has to hold off on optimising not frequently called functions.
There are also often multiple concrete types that can be passed in, optimising for one will not help if it is also getting called with other concrete types.