Hacker News Reader

Comment by trolan | Hacker News Reader

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects.

Once the vectors are created tho, you're completely off the cloud if you so choose.

You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes.

Everything else is already using huggingface here and can be swapped out for any other model besides GPT2 which supports the prompts.

View on HN

> Once the vectors are created tho, you're completely off the cloud if you so choose.

Ehr no? You'll need to also create an embedding of your query, which makes you totally dependent on OpenAI. If you swap out embedding algorithm you will have to regenerate all the embeddings as well, they might not be even the same size.

Ah, I see what the top comment was implying. I was a bit short sighted on that side. Yes, you'd be tied to OpenAI for any new queries you need to generate. There could be some ways to offload that (vector of a vector) but it is a cloud dependency. I'd argue not a cost dependency based on how cheap these are.

Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?

The only embeddings I currently see listed on https://openai.com/pricing are Ada v2, at $0.1/million tokens.

Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that cost of time matches the UN abject poverty level. This is also why it's never been the year of linux on the desktop, and why most businesses still don't use Libre Office and GIMP.

I can't speak for quality; even if I used that API directly, this whole area is changing too fast to really keep up.

If you look at a embeddings leaderboard [1], one of the top competitors called InstructorXL [2] is just a pip install away. It's neck and neck with Ada v2 except for a shorter input length and half the dimensions, with the added benefit that you'll always have the model available.

Most of the other options just work with the transformers library.

[1] https://huggingface.co/spaces/mteb/leaderboard

[2] https://github.com/HKUNLP/instructor-embedding

Setting up an embedding alternative out of huggingface sentence transformers is fairly easy, the magical thing openai does is that they will create embedding of 8192 characters at a time while most other emerging will force you to chunk your documents in 512 characters long sequences, losing lot of context, multiplying your queries result, search times etc

If you've never coded or used Python before, yeah, go with OpenAI. Otherwise, generating embeddings with SentenceBERT takes 5 minutes.

And from my personal experience Ada embeddings are not the best. They are large (makes aproximate searching harder), are distributed weirdly, and zimply put, other embeddings give better results for retrieval.

Another advantage is that you are not an OA's whim: they just announced the deprecation of some previous model. What are you going to do when they will deprecate Ada v2 and you've built a huge system on top of it? You'll have to regenerate embeddings and hope everything still works just as well.

Yes, exactly this, I also want to say I'm not someone who generally thinks open models are better. I think embeddings just haven't been a focus for OpenAI and it shows. Maybe in the future they will focus on it

Yes, when new technology comes you may need to upgrade. OpenAI aren't hero's but they are covering the cost to move people from old to new embedding models

Running models on your own hardware isn't just about cost, there are privacy concerns too.

No, I have anecdotal evidence using it. How does it compare in your usage of OpenAI and competitors?

I'm interested if you have specific options you think are better and/or cheaper.

Can you elucidate on what those signs are? Thanks in advance

As per the Massive Text Embedding Benchmark (MTEB) Leaderboard maintained by Huggingface, OpenAI's embedding models are not the best.

https://huggingface.co/spaces/mteb/leaderboard

Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).

What? It's only one file, and it definitely looks like it's using openAI to make the actual queries.

    qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())

You can change the arguments to from_llm() to point to a local model instead. Example here: https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discuss...

Reply on news.ycombinator.com