Hacker News Reader

Comment by porridgeraisin | Hacker News Reader

> Unlike full language models, I don't think the primary business of Typesafe will be serving Jev at API pricing; it'll look a lot more like putting Jev at the center of a much more expensive suite of software.

Precisely this. Should be top comment.

Also, the model moat is understated as training data for these purposes also accrues to the winner, which due to the first mover advantage as well as the distribution advantage you speak of, is typesafe. In contrast to relatively open coding data. Openai anthropic also have that, but like you say its a different business.

View on HN

The are still just 1tok output of pretty standard llms just along with the logprobs converted to some json

At least in theory (TypeSafe has been pretty close-lipped about the details so this might just be hot air, and I think the evidence is a bit spotty) this is false, since they use a different reinforcement training method.

If the word “calibration” in probability doesn’t mean anything to you, the difference isn’t very apparent, but that doesn’t mean it doesn’t exist.

> If the word “calibration” in probability doesn’t mean anything to you, the difference isn’t very apparent, but that doesn’t mean it doesn’t exist.

I am really sceptical that they've developed a zero shot calibrated classifier for arbitrary inputs.

That being said, if they have, then this valuation is like 100 times too low.

It's calibrated where it's calibrated and not where its not. Their "confidence" score is rather bunk, so it's hard to say. As with other LLMs, you have to eval it for your setting, and if you find it calibrated, then great.

Interestingly enough, at lunch this week it turns out that one of my buddies work colleagues has actually developed a relatively well calibrated version of this, so it's not impossible.

It's probably much harder for arbitrary data though (like most things).

> Their "confidence" score is rather bunk, so it's hard to say.

Then it's not calibrated (in the technical sense of the term).

Reply on news.ycombinator.com