Hacker News Reader

Comment by garciasn | Hacker News Reader

We host our own LLM because:

1. We are not permitted, by rule, to send client data to unapproved third parties.

2. I generally do not trust third parties with our data, even if it falls outside of #1. Just look at the hoopla with Zoom; do you really want OpenAI further solidifying their grip on the industry with your data?

3. We have the opportunity to refine the models ourselves to get better results than offered out of the box.

4. It's fun work to do and there's a confluence of a ton of new and existing nerdy technology to learn and use.

View on HN

Stupid question from an outsider, but is it possible to grab a pretrained model (most likely one of the "camelids"), feed it your own data (I have about 1k documents that cannot leave my network) and use it for fast-but-sometimes-wrong information retrieval?

What is your experience with quality? Even if you don't have the option to use a third-party LLM the question is if your self-hosted solution is good enough so your users (employees) will accept it. While you can forbid external solutions in the end you can't force them to use your own solution - at least not in the long run.

I'm very curious what your experience is? Do you think to self-host is good enough so users will accept it?

1. We can and do disallow sharing of client and company data with third party LLMs by policy and, yes, we can ‘force’ them to use our product capabilities.

2. We tested 13 and 30b parameter pretrained LLMs and found them “good enough” within the confines of the original scope.

That said, we are expanding the use cases and with that comes the need for increasing levels of quality. This is where the self-training comes in.

We’re fairly confident it’ll meet our needs for now.

Thanks for your answer. I think I have to research this further. I disagree with your point about forcing your employees. Everyone has GPT as a comparison, if the internal solution is a little bit worse it might fly. If it is too bad people will not put up with it and if you force them they will either ignore the internal solution or leave.

EDIT: Or in the worst case ignore the policy and get fired.

Getting fired could be the least of their concerns depending on the nature of the data shared with third parties.

I think a good example is Microsoft Source Safe. Microsoft had the strongest incentive to use their own product internally and yet even a company this size could not resist the pressure in the long run to use a marginally better product.

We will see how this turns out with LLMs and while I wish companies were more concerned about what data they share with third parties my experience is that apart from military/intelligence related data everything can and will be outsourced.

Reply on news.ycombinator.com