Hacker News Reader

hellovaion Classifying customer messages with LLMs vs traditional ML

That's very interesting! What sort of direction did you head in with prompt optimization? Was it mostly in shrinking it and then using multi-shot examples? We found that shorter prompts (empirically) perform better than longer prompts.

hellovaion Classifying customer messages with LLMs vs traditional ML

SetFit is fairly good, and we do help train SetFit like models for the results you get, however, the issue with SetFit is that its latency and cost benefits come at the price of flexibility.

If you want to add a new class, to update an existing one, it requires training a new model. Sometimes this is ok and sometimes it's not. This is why we generally prefer a hybrid approach where some classes are using traditional models (BERT based) while others are determined by the LLM.

hellovaion Classifying customer messages with LLMs vs traditional ML

Thanks Alex, in this article we focused more on deployment comparisons, for example the cost and latency of what it would take to deploy a BERT based model vs LLMs.

In a future article, we're planning on posting accuracy comparisons as well, but here we want to evaluate a few other architectures for comparison. For example, at 1TPS with 1k tokens, chat-gpt-turbo would cost almost $5k vs a simpler BERT model you could run for under $50.

This is probably very obvious to some people, but a lot of people's first experience with any sort of AI is often an LLM, so this is just the first of many posts we hope to share.

hellovaion Classifying customer messages with LLMs vs traditional ML

oh! Good catch! Fixed this, and will update in the release.

hellovaion Classifying customer messages with LLMs vs traditional ML

That's a good point! We're actually working on integrating this as well, but in practice, what we've found is that LLM's in general don't like to respond with empty strings for example.

My hypothesis here is that due to RLFH, there's likely some implicit learning that tangentially related content is better than no content.

Given that, you'd likely still get better results with your schema being:

"string | null" so the LLM can output a null instead of "" since there is probably not as much training data that gives "" high log prob values.

But we're looking forward to evaluating the functions call, and seeing what the metrics show!