Hacker News Reader

Am i reading it right that the fine tune a model using 20 examples and 5 epochs? That seems really weird for me

LLMs are few shots learners, that's why many people put examples into prompt, this is the next step.

I don’t believe few shot performance dictates how quickly you can fine-tune.

Most fine tunes will have much larger datasets (I am under the impression you want 10’s of thousands of examples for most runs).

So I’m similarly impressed 20 examples would make such a big difference.

But also note entity density decreases as example count increases. This is counterintuitive — maybe something else is going on here?

usually higher parameter models do better with less training data, seperate from few shot learners, but related in other ways.

https://github.com/huggingface/setfit gets good fine-tuned scores on some downstream tasks with just 8 labeled examples.

Can't overfit when your learning rate is zero! insert smart thinking meme