Hmm maybe. Turns out the author trained a logistic-regression classifier on the embeddings too, but didn't report the results:
https://github.com/thelgevold/fine-tuned-classifier/blob/mai...
https://github.com/thelgevold/fine-tuned-classifier/blob/mai...
In summary: Using logistic regression actually improves accuracy, but also performance during both runtime and during training.
Do 5-fold cross validation, maybe stratified.