Show HN: Dpq – a small Python library to process data using LLMs

github.com

4 pointsNaa41 comment

Hi everyone! I wrote a small python library to simplify data processing and feature engineering. I was inspired by a recent video on sentiment analysis using LLMs by Andrew Ng and tried to come up with a convenient way of applying this logic to data frames for data folks like myself. Excited to hear what everyone thinks!

Open articleView on HN

Not familiar with similar libraries so not sure how this stacks up. But a few things that I see as helpful improvements would be: 1. using templating engine like jinja to allow for more flexibility in customizing prompts 2. sending multiple rows per request. If I want to do sentiment classification, I may have a fairly large bit of text explaining exactly what I want the classifier to do. If I include that instruction along with every individual sentence I want to classify, I'm wasting a lot of tokens. Instead I'd rather send the instructions along with 100 sentences and ask it to classify each one. This sort of functionality would work well with what you're trying to accomplish using dataframes.

Reply on news.ycombinator.com