Hacker News Reader

Comment by chaxor | Hacker News Reader

The most frustrating thing about the many, many clones of this exact type of idea is that pretty much all of them require OpenAI.

Stop doing that.

You will have way more users if you make OpenAI (or anything that requires cloud) the 'technically possible but pretty difficult art of hoops to make it happen' option, instead of the other way around.

The best way to make these apps IMO is to make them work entirely locally, with an easy string that's swappable in a .toml file to any huggingface model. Then if you really want OpenAI crap, you can make it happen with some other docker secret or `pass` chain or something with a key, while changing up the config.

The default should be local first, do as much as possible, and then if the user /really/ wants to, make the collated prompt send a very few set of tokens to openAI.

View on HN

It's difficult to compete. A small business might answer 10,000 requests to their chat bot. The options are

- Pay openai less than $50mo

- Manage cloud gpus, hire ml engineers > $1000/mo

- Buy a local 4090 and put it under someone's desk, $no reliability +$1500 fixed

Any larger business will need scalability and you still can't compete with openai pricing.

Maybe one of you startup inclined people can make an openllama startup that charges by request and allows for finetuning, vector storage

I’ve got an expensive GPU at home I’m not even using because there aren’t that many things to do with it. Give me more local options.

https://github.com/oobabooga/text-generation-webui

https://github.com/bentoml/OpenLLM

https://www.reddit.com/r/LocalLLaMA/top/?t=month

And

https://github.com/go-skynet/LocalAI

https://github.com/juliooa/secondbrain

https://github.com/louisgv/local.ai

https://faraday.dev/

Let other people pay you to run their stuff on your hardware with Vast.ai.

Even if you are not into coding there are many good AI tools that run local. Two very easy examples:

I've had great fun with the " Easiest 1-click way to install and use Stable Diffusion on your computer."

https://github.com/easydiffusion/easydiffusion

And while Whisper is OpenAI, it is trivial to use locally and extremely usefull

https://github.com/chidiwilliams/buzz

It depends heavily on the use case, not org size. I consult for a ~70 people org that needs to process ~1M tokens per day. That costs $30K per day on OpenAI ChatGPT API. I'm sure this is not an extraordinary case.

Each person in the org needs 1M GPT-4 token and semantic search can’t be used to trim queries? I would be super curious to know more about this use case.

The data doesn't scale according to employee size. If they manage to cut the headcount in half, they'd still need to process the same amount of info.

The use case is based on public information on the internet. News articles, PRs, social media posts, etc.

LLMs are used to extract info from text in a structured format. It used to have several classification and NLP models to do the job, but now a single LLM can do it faster and with better accuracy.

I have a 4080, let’s do a startup. #cancode #hashomelab

> Maybe one of you startup inclined people can make an openllama startup that charges by request

I'm currently building www.lalamon.us specifically to provide a fully hosted open source model experience. One slight difference is that I'm providing a private chat instance for each user, so charging based on hours of active chat usage seemed to make more sense. Per-request charging seems more unpredictable for users, but I'd be interested in hearing the case either way.

Feel free to reach out with more questions if interested; my email is in my profile.

Doing this. We soft launched yesterday with a paid Falcon-40B playground - 3 models for now Falcon 40b instruct, uncensored, and base. Adding API and per token pricing this week.

https://api.llm-utils.org/

And more models coming soon.

Vector storage isn’t on the roadmap (what stops using a separate vector store from working well? Could add to roadmap but want to add understand more first), and we could add fine tuning if it’s a common request.

Lots of people using LLMs to make chat bots from their existing datasets: customer service troubleshooting, FAQs, billing, scheduling. Being able to upload their own pdfs, spreadsheets, docx, crawl their home page, lets the chat bot become personalized to their use case. While you could locally query your own vectordb before prompting, people buy paid service so they won't have to manage any of the technical details.

If people can drag and drop some files from their nas, you parse them with apache tika or similar https://tika.apache.org/ , they can start using personalized branded bots. It also lets you do things like refusing to answer, if the vector database returns nothing and the use case requires a specific answer from the docs only (not the llm to make stuff up).

For those use cases the “custom ChatGPT” tools I linked here might be better https://news.ycombinator.com/item?id=36649777

Shouldn't you use a .com tld?

Will your pricing be competitive with Replicate?

Not secure... NET::ERR_CERT_COMMON_NAME_INVALID Subject: *.safezone.mcafee.com

Issuer: McAfee OV SSL CA 2

Expires on: Aug 3, 2023

Current date: Jul 8, 2023

PEM encoded chain: -----BEGIN CERTIFICATE----- MIIGfzCCBWegAwIBAgIQKt9VNrFtaozA1bILX1OcfzANBgkqhkiG9w0BAQsFADBk MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0

FastChat-T5 can work for such a use case and it runs on (beefy) CPUs. With a 700$/month instance, it can do 4 conversations simultaneously, without needing GPUs.

The instant a company has sensitive data, this becomes very viable.

Wait until winter time and heat your house!

Good double use of that low entropy energy. Heat pumps excepted.

People don't scale. This is personal. Only 3 is a good choice for people in a site with the name hacker something.

What do you (or anyone else, feel free to chime in) do with other LLMs that makes them useable for anything that is not strictly tinkering?

Here is my premise: We are past the wonder stage. I want to actually get stuff done efficiently. From what I have tested so far, the only model that allows me to do that halfway reliably is GPT-4.

Am I incompetent or are we really just wishfully thinking in HN spirit that other LLMs are a lot better at being applied to actual tasks that require a certain level of quality, consistency and reliability?

I still wonder what makes GPT-4 so much better than its contemporaries. That's why I saw tons of people trying to explain how GPT-4 works starting from simple neural network distasteful, tons of people already knew and do that but none of them is nearly close to GPT-4.l

> I still wonder what makes GPT-4 so much better than its contemporaries.

OpenAI have had many years to craft their dataset down from the noisy public datasets, and GPT4 is (supposedly) a mixture of 8 "expert models" each of which is 220B (5x+ larger than the Falcon 40B) with a total of 1.7B parameters (3x+ Google's huge 540B PaLM). The hardware and software to train networks of that scale is also a deep moat. Relatively speaking, the model architecture ("gpt from scratch") is the easiest piece.

From my understanding. GPT-4 is the biggest, or one of the biggest. It was trained on low quality internet datasets, like the others. What makes it different is post-training on custom data with human supervision. We know they even outsourced that to Africa. Second, they integrated it with external tools. Like Python interpreter, internet browser. But the first is most important. Also most likely they have experimented and found some tricks which make it bit better.

They pay tons of people to type out conversations that they can feed into it. It's just a lot of people doing a lot of work.

This line of thinking only works if it's impossible to imagine a world where OpenAI isn't the leader. In 2 years if the non OpenAI models are better then it will serve us much better to allow these tools to work with other models as well.

Since OpenAI is all just APIs with simple interfaces, I don't think that plugging a different, capable model in whatever tool you are building is going to be an issue.

You are correct in this assesment. A majority of individuals and startups playing around with turning LLMs into products aim to be prepared for the arrival of the subsequent generation of models. When that occurs, they'll already have a product or company in place and can simply integrate the new models.

Models are getting commoditized, well executed ideas are not.

> is that pretty much all of them require OpenAI.

They're not here to release an actual product. They're here to release part of a CV proving they have "OpenAI" experience. I'm assuming this is the result of OpenAI not actually having any homegrown certification program of their own.

> OpenAI not actually having any homegrown certification program

A bit off topic but where are certifications (e.g. Cisco, Microsoft) useful? I am sure they are useful (both to candidates and companies) because people go to the effort to get these certs, and if they were useless everyone would have stopped long ago. I don't assume people do it for ego satisfaction.

But I've never worked anywhere where it has come up as an interview criterion (nobody has ever pointed it out when we are looking at a resume, for example). Is it a big business thing? Is it just an HR thing?

Years ago, companies could get discounts if they were a “certified gold partner” or whatever.

To be a partner, the company would need a certain number of certifications among their employees, so there was tangible value to companies who either used a ton of Microsoft licensing or Cisco/Dell hardware, or resold those to their own clients (better discount equating to higher margin).

In some cases, getting the higher level certifications like Cisco CCIE was a virtual guarantee of a good job.

I feel like this has become less of a thing in recent years, but I’m not involved in that space anymore.

Definitely still a thing with Azure and Atlassian

Could you elaborate or drop some links? Are people getting discounts on Azure cloud bills with certifications?

It is a consulting / business partner thing. Different levels in the business partner programs require minimum number of certified employees in your consulting firm. So if you work in that slice of the industry, certifications matter. Outside of that... not so much.

Mainly when applying for a corporate job where you have 0 referrals. It's a guidepost that you at least have some idea what you're doing and are worth interviewing when people can't find someone who knows you and your previous work.

I've only ever seen it as a people who don't have a job thing.

https://mudler.pm/posts/smart-slackbot-for-teams/ with LocalAI?

This is a well-written tutorial and it's exactly what I was looking for! Thanks so much.

The only OpenAI 'crap' being used here is to generate the embeddings. Right now, OpenAI has some of the best and cheapest embeddings possible, especially for personal projects.

Once the vectors are created tho, you're completely off the cloud if you so choose.

You can always swap out the embedding generator too, because LangChain abstracts that for your exact gripes.

Everything else is already using huggingface here and can be swapped out for any other model besides GPT2 which supports the prompts.

> Once the vectors are created tho, you're completely off the cloud if you so choose.

Ehr no? You'll need to also create an embedding of your query, which makes you totally dependent on OpenAI. If you swap out embedding algorithm you will have to regenerate all the embeddings as well, they might not be even the same size.

Ah, I see what the top comment was implying. I was a bit short sighted on that side. Yes, you'd be tied to OpenAI for any new queries you need to generate. There could be some ways to offload that (vector of a vector) but it is a cloud dependency. I'd argue not a cost dependency based on how cheap these are.

Do you have citations on OpenAI embeddings being some of the cheapest and best? The signs I've seen points almost in the opposite direction?

The only embeddings I currently see listed on https://openai.com/pricing are Ada v2, at $0.1/million tokens.

Even if the alternative is free, how much do you value your time, how long will it take to set up an alternative, and how much use will you get out of it? If you're getting less than a million tokens and it takes half an hour longer to set up, you'd better be a student with zero literally income because that cost of time matches the UN abject poverty level. This is also why it's never been the year of linux on the desktop, and why most businesses still don't use Libre Office and GIMP.

I can't speak for quality; even if I used that API directly, this whole area is changing too fast to really keep up.

If you look at a embeddings leaderboard [1], one of the top competitors called InstructorXL [2] is just a pip install away. It's neck and neck with Ada v2 except for a shorter input length and half the dimensions, with the added benefit that you'll always have the model available.

Most of the other options just work with the transformers library.

[1] https://huggingface.co/spaces/mteb/leaderboard

[2] https://github.com/HKUNLP/instructor-embedding

Setting up an embedding alternative out of huggingface sentence transformers is fairly easy, the magical thing openai does is that they will create embedding of 8192 characters at a time while most other emerging will force you to chunk your documents in 512 characters long sequences, losing lot of context, multiplying your queries result, search times etc

If you've never coded or used Python before, yeah, go with OpenAI. Otherwise, generating embeddings with SentenceBERT takes 5 minutes.

And from my personal experience Ada embeddings are not the best. They are large (makes aproximate searching harder), are distributed weirdly, and zimply put, other embeddings give better results for retrieval.

Another advantage is that you are not an OA's whim: they just announced the deprecation of some previous model. What are you going to do when they will deprecate Ada v2 and you've built a huge system on top of it? You'll have to regenerate embeddings and hope everything still works just as well.

Yes, exactly this, I also want to say I'm not someone who generally thinks open models are better. I think embeddings just haven't been a focus for OpenAI and it shows. Maybe in the future they will focus on it

Yes, when new technology comes you may need to upgrade. OpenAI aren't hero's but they are covering the cost to move people from old to new embedding models

Running models on your own hardware isn't just about cost, there are privacy concerns too.

No, I have anecdotal evidence using it. How does it compare in your usage of OpenAI and competitors?

I'm interested if you have specific options you think are better and/or cheaper.

Can you elucidate on what those signs are? Thanks in advance

As per the Massive Text Embedding Benchmark (MTEB) Leaderboard maintained by Huggingface, OpenAI's embedding models are not the best.

https://huggingface.co/spaces/mteb/leaderboard

Of course, that's far from saying that they're the worst, or even headed that way. Just not the best (those would be a couple of fully opensource models, including those of the Instructor family, which we use at my workplace).

What? It's only one file, and it definitely looks like it's using openAI to make the actual queries.

    qa = ConversationalRetrievalChain.from_llm(OpenAI(temperature=0.1), db.as_retriever())

You can change the arguments to from_llm() to point to a local model instead. Example here: https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML/discuss...

ClosedAI has freaked me out with how much power they have, and how irresponsible they are with it.

I'm so horrified that they are going to take away the ability to ask medical questions when the AMA comes knocking at their door.

I don't liked closedAI either, this seems like the first tech I've played with in a long time that was great on day 1, and seems to get progressively less great over time.

Amen, local first should be the default for anything that sucks all my data.

Although until these things can do my laundry none of them deserve any of my compute time either.

txtai makes it easy to use Hugging Face embeddings and Faiss, all local and configurable. https://github.com/neuml/txtai

paperai is a sub-project focused on processing medical/scientific papers. https://github.com/neuml/paperai

Disclaimer: I am the author of both

Honestly, for me, getting good quality results matters way more than keeping my searches private. And for that, nothing compares with GPT4.

Have you seen PrivateGPT. It's quite good and free.

It's not nearly usable. It's functional in that it spits out a response. Can that response be used for anything useful? No.

what hardware do you need for that?

Consumer-grade, AFAIK it's GPT4All with LLaMA.

That's doesn't really answer the question.

Neither does the github "System requirements" section, which I find disappointing. Ideally, it should give minimum memory requirements and rudimentary performance benchmark table for a sample data set, across a handful of setups (eg, Intel CPU, Apple M1, AMD CPU, with/without a bunch of common GPUs). With that information I would know whether or not it's worth my time even trying it out on my laptop.

Edit: lol, I went through 2 pages of the issues on the github page and most of them could be avoided by putting this basic information into the system requirements:

https://github.com/imartinez/privateGPT/issues/174 https://github.com/imartinez/privateGPT/issues/179 https://github.com/imartinez/privateGPT/issues/104 https://github.com/imartinez/privateGPT/issues/141 https://github.com/imartinez/privateGPT/issues/282 https://github.com/imartinez/privateGPT/issues/316 https://github.com/imartinez/privateGPT/issues/333

... And lots more! Some of these people have 128gb memory and 32 cores, and still find it "very slow". Others having memory pool errors. Some of the answers hand-waving at needing "a more better computer"

I reckon a lot of these issues could be closed and linked to a single ticket for proper hardware requirements in the readme.

Link?

Presumably https://github.com/imartinez/privateGPT

It runs (slowly) on my 6 year old i5 laptop.

> Stop doing that.

This commanding attitude on HN seems to be getting worse lately. Not a fan.

#Chaxor: I fully agree with this. I am not keen on this being a one horse race, and for privacy reasons would like to deploy these models locally. However, it seems for many programmers it is somewhat easy to build something that can query into OPenAI so they can put it on their resume.

Do you know of any FAISS / open source / one-click install w<windows app here I can search in my PDFs via vectors? I can see Secondbrain.sh will have the function in the future, but currently it does not.

I have around 500 documents I want to be able to search in.

When I was creating this tool, I made sure to abstract out the reliance on just one LLM or vector db. Instead, I focused on using langchain/huggingface for tokenization, embedding, and conversational modeling. This was done purposely so that it would be simple to replace the OpenAI dependencies with any other LLMs if needed.

Even GPT-3.5-turbo-16K isn't good enough for most retrieval augmented generation tasks.

Locally ran LLMs are far worse.

I don't like it either, but for now, if you want good RAG, you have to use GPT-4

Gpt4all does exactly that. You can choose between local model or bring your own openai token.

Does it provide a uniform interface that includes: retries, caching, streaming?

Reply on news.ycombinator.com