Hacker News Reader

Comment by fswd | Hacker News Reader

I could add a couple things from my own experiences. Storing prompts in a database seemed like a good idea, but in practice it ended up being a disater. Storing the prompt in a python/typescript file, up front at the top, works well. Using OpenAI playground with it's ability to export a prompt works well, or even better, something in gradio running in vscode with debugging mode, works even better. Few shots with refinements works really well. LangChain did not work well for any of my cases, I might go as boldly as saying that using langchain is bad practice.

View on HN

It's delightfully hacky, but we actually have our prompt (that we parameterize later) stored in a feature flag right now, with a few variations! I actually can't believe we shipped with that, but hey, it works? Each variation is pulled from a specific version in a separate repo where we iterate on the prompt.

We're going to likely settle on just storing whatever version of the prompt is considered "stable" as a source file, but for now this isn't actively hurting us, as far as we can tell, and there's a lot of prompt engineering left to do.

https://thedailywtf.com/articles/the-inner-json-effect

So much "Yes!" for LangChain being bad practice. An unnecessarily bloated abstraction over what should be simple API calls.

Do you have a recommendation of how to easily connect a language model to a python repl, apify, bash shell, and composable chaining structures if not langchain? I find those structures invaluable but am curious where else I could build these programs.

It’s great for prototyping and seeing what is possible but for running in production you’ll likely need to write it yourself, and it will just take a few minutes.

Could be we're in a (short?) interregnum analogous to pre-Rails Ruby: there are lots of nascent frameworks, but the dominant one hasn't been born yet. FWIW - DIY is working well for me.

Check out https://www.featurebase.com/blog/doctorgpt-harnessing-the-po... and the DoctorGPT repo for doing some things with documents and LLMs, without a framework. I use simple templating and vectors to assemble prompts.

The current trend for productionizing LLM-based applications is to write your own (really thin) wrappers around the actual LLM call. The majority of your code should be business logic independent of the LLM, anyway: information retrieval, user interface, response logging, and so on.

In my experience with cloud-based and local models, LLM chaining compounds errors; I would urge you to look at few-shot, single-query interaction models for business applications using LLMs as a new unit of compute.

Could you explain how storing prompts was a disaster?

Prompts should be really easy to cut and paste from an editor to a playground. Updating a database is unnecessary friction with no real benefit.

I’m keeping all our prompts in a json file (along with some helpful metadata for us humans).

No idea if I’m doing it right.

Reply on news.ycombinator.com