Would that even work? Using the pretrained model, wouldn’t you then be in the “local” optimum?

Although from a transfer learning perspective, chopping off last layers and retraining… You are prob right