Even before LLMs the frameworks for “end-to-end” development of ML models have sucked including the ones I’ve developed ay my employers and the ones I have developed for myself.
What I’m talking about is the need to do
https://scikit-learn.org/stable/model_selection.html
which is not just about training one model but also about evaluating at least one model and usually generating quite a few models with different parameters the latter of which is more essential than ever now that we don’t really do early stopping with neural networks anymore. (Back in the day I built products that could reliably build a useful model in one try with early stopping, which made fine-tuning LLMs so nerve-wracking to me)
For instance the tools I mention above in scikit-learn work great if I am training models that take a few minutes to train on one machine. I have a set of sk-learn models that I train and use every day.
I’ve tried fine-tuning models from huggingface to do the same task, on a good day they perform about as well but take 30 minutes to train instead of 30 seconds. I really ought to be able to run both kinds of models in the same “model selection” framework and pick the best, but boy is it a hassle because it not just about model selection it is about the whole data handling and training process which runs inside the model selection process. For my problems I don’t really like the way huggingface handles data as much as sk-learn/pandas/bumpy and don’t like the model selection facilities in huggingface.
I worked at a place where our model trainer would train foundation models (a few days on a specialized machine, could just as easily have been a swarm of cloud GPU instances) and also train models that incorporated foundation models (a few minutes to train.). We developed our own model selection facilities that worked in the large and at the time I didn’t appreciate the facilities in skl, what we built wasn’t that good and I take 100% of the blame that can attributed in our company to that but the fact that every other framework in the industry sucks (ok, i like SKL for small problems but for large problems it just doesn’t show up)
I’ve built my own code which approximates some of what langchain does, I wouldn’t say it “sucks” the way langchain does but it is very simple, very targeted at one problem, if I tried to grow it up like langchain I’d be swearing at it to.
This discussion really caught my eye because it shows how LLM training is still pre-paradigmatic. I’ve seen this problem and thought I was alone
https://news.ycombinator.com/item?id=37399873
but now I know I am not. Situations like that leave me, as an engineer of practical solutions, feeling really out of control, but academics don’t seem to mind so much because they just need numbers for their conference paper. What gets me is that so many people were seeing this problem and not writing up although it’s pretty deeply disturbing and makes it look like we are training, training and training our models and most of it is wasted work.