depends, but you fine tune the whole thing, check ulmfit paper and fastai libs