Hacker News Reader

Comment by mhitza | Hacker News Reader

> How does Orbit work?

> Orbit is a Firefox add on that uses AI to summarize and answer queries about web content such as articles and videos.

> When a user asks Orbit to summarize or query content, Orbit gathers the context (eg. text, images, videos, etc.) of the page the user is viewing and provides a summary or answer. Orbit works on websites including Gmail, Wikipedia, NY Times, YouTube, and more.

> For the current version, we are using a Mistral LLM (Mistral 7B) hosted within Mozilla’s GCP instance.

It's interesting they're going with Mistral 7B. Is anybody else using Mistral 7B in production? And in what role?

I've considered using it for general knowledge type questions, and as a way to classify information, but would have never considered it for summarization type tasks due to it's limited context size (8k).

View on HN

Mistral 7B has a context window of 32k. I use it in production for medical summarization tasks supporting appeals and physician advisory services.

Why are you confident to use such a tiny model on something so critical?

I would never use anything smaller than a 70B model for anything even vaguely medical related!

Its summarisation, who cares if its right as long as you feel confident after reading it? /s.

In my experience, even GPT-4o is terrible at revealing information from things longer than a few pages.

It might be an issue with dimensionality reduction in general though. If you think about it, you can't really take away much of what is contained within any given amount of text with text, unless the source was produced extremely inefficiently.

Producing outlines or maybe a form of abstract, it seems to be okay at, but you would never really know where it fails unless you read the entirety of the source text first to begin with. IMO, its not worth risking unless you plan to read the source anyway or its not really important.

Try to walk through a Wikipedia article having an LLM summarize every few paragraphs, its often wildly inaccurate.

A key thing to remember in this specific moment in history is that the vast majority of people will be as lazy as they can get away with being. People want to leave work and go home and if an LLM lets them do that faster, who cares if it's accurate? It can absorb the blame.

> Why are you confident to use such a tiny model on something so critical?

Don't use any LLM for something critical. They can't be trusted innately due to their design, why on earth would you use them for something where you need reliability?

> why on earth would you use them for something where you need reliability?

As very best I can tell the answer is "because bags of cash"

I don't use Mistral 7B alone, this is just a component in a RAG-based system. A system that is 1) not clinical facing, 2) not used in clinical decision making, 3) provides in-line references sources for end users to validate information, and 4) is inherently human-in-the-loop.

Hope your prod is being done currently in this political... Climate.

Reply on news.ycombinator.com