If it's a fine tuning step at the end, why is the need for it to vary a problem? Can't you run the fine tuning, test for regression, and deploy the weights in a day?

I think the more likely reason is it doesn't work as well as in context learning. Otherwise they would prefer to avoid polluting context and degrading performance.