I tried fine-tuning BERT models for a content-based recommender system and found they used more than 50x the effort to train than an embedding based model that performed about as well.

Fine-tuned BERT failed completed at predicting if a headline would get upvoted on HN or what the comments/votes ratios would be. Those are even noisier than the recommender so it comes as no surprise to me. (For both of those my best model is still a bag-of-words model.)