Hacker News Reader

Comment by psnehanshu | Hacker News Reader

I think for this purpose a small model will do reasonably well

But in some cases "reasonably well" is not good enough when you can get a "very good" or "excellent" by stepping up a model category.

I tried this with an AI subtitle translator thing I built for myself. Mid-tier models were considerably cheaper but "reasonably good" doesn't cut it in translations, the output language was too janky or used the wrong words. I had to bump it to Sol level (at the time) to get "good / very good" output, which was enough for something I intended to use once.

What's weird is I didn't see mention of accuracy in the article.

Not really. Haiku is really not good at decision tasks (source: real experience).

What about Haiku 5.5?

Right. So we can either assume the engineers at Amazon are fucking clowns who don't do any engineering, or we can assume they tested and found that Sonnet was the cheapest model that met their reliability targets.

If you think the former is a likely hypothesis, then why did you even click on TFA?

Let's keep the discussion civil

One can think the circumstances a circus without thinking the participants all to be clowns.

As for your question, seeing what others are doing helps one decide both what to do and what not to do.

Good points, agreed.

Anthropic was the only frontier lab on bedrock for a while. Hate to break it to you but there are in fact many scenarios with reasons and pressures for non engineering-driven decisions to dominate

But the question is also "Would Haiku not have worked?". If they ruled that much out, then they aren't going to be using anything super cheap. Also, they already had been hosting some of those cheap models on Bedrock and I'm sure they use them where it makes sense.

That's rather extreme? There is an possibility than Amazon's engineers are neither infallible nor idiots. Regardless of this specific situation highly talented engineers very often take decisions that lead to suboptimal outcomes on organizational level due to various reasons.

It's also possible this was optimal on the organizational level because of opportunity costs. You may be wasting $2M on inefficient compute but if your other opportunities are higher ROI and you're not allowed to add headcount then it sits. Disclaimer: At Amazon but no inside knowledge of this specific situation

Reply on news.ycombinator.com