Hacker News Reader

Comment by 6gvONxR4sf7o | Hacker News Reader

I've done a lot of work on information extraction with these over the last year, and if accuracy counts, then a) GPT4 is in a league of its own, and b) GPT4 still isn't really very good. They may not have a "moat," but they're still the only player in town when quality is critical.

View on HN

For now. The quality of competitors has been improving considerably when we look at our own in-house analysis for various use cases we have.

It looks like GPT4 has approached an asymptote in quality (at least within a compute time window where they remain even marginally cost effective). Others are just catching up to that goalpost.

Even GPT4 suffers from the same problems intrinsic to all LLMs-- in real world use, hallucinations become a problem, they have a very difficult time with temporal relevance (i.e identifying when something is out of date), and they are horrifically bad at any kind of qualitative judgement.

> They may not have a "moat," but they're still the only player in town when quality is critical

Their initial moat was built with ChatGPT, which was launched less than a year ago and was surpassed by competitors in less than 6 months. Their current GPT4 is less than 6 months old. While your statement may be true for now, I don’t expect it will hold longer term. They have name recognition advantage, but so did AOL.

ChatGPT wasn't surpassed by competitors in less than 6 months, what model would you say beat it that early?

I think they mean gpt3.5 ChatGPT

Correct, there are a number of models available that perform similarly to GPT3.5 for the majority of tasks an end user may ask of it.

> a) GPT4 is in a league of its own, and b) GPT4 still isn't really very good.

Agree with both and I work in information extraction too.

what is information extraction in your terms? sounds like some spy stuff

Input some document, get a JSON with all the fields. It requires understanding a lot about world entities, fields and values to parse free form into structured form. Also works on screens, you can extract information from an app, for example to run an AI bot on top of the app.

My use cases were extracting stuff like this from scientific papers: "what were the interventions in this study, what outcomes were measured, and what was the treatment effect of each intervention on each outcome?"

Probably a prompt like “read this and tell me the wether it is about a stock and wether i should buy or sell based on the article”

Not OP but I work in a similar space.

Most likely parsing unstructured data (a superset of NER).

Did you try Falcon 180B?

Nope, but from TFA it looks like it's barely better than llama 2, which is significantly worse than GPT4.

Reply on news.ycombinator.com