hbrn

It sounds like the problem isn't really that engineer's optimizations are premature. The problem is that engineers have an opinion about the code that doesn't really matter.

What I think you're saying is that opinions that LLMs produce are good enough and malleable, so why even have an opinion?

Which is debatable - oftentimes their opinions are not good enough, and not malleable at all.

We're all still learning how to distinguish those. But in order to learn this, you have to read and understand the code they produce. LLMs can migrate a codebase to Rust, but they still suggest walking to a car wash. Every day they make decisions that humans would be fired for.

LLMs are definitely not yet capable enough to blindly trust them, and there's a pretty good chance they never will be.

It's still going to be a little worse than frontier models, and for some people that can be enough to pay $36k/year.

Oh and prefill is still a lot worse on local LLMs. Like 1-2 orders of magnitude worse.

> Where is all the great software? Really, where?

It exists, at least in terms of ROI. Some enterprises are able to do successful rollouts that allow them to significantly shrink workforce in certain departments without losing quality (no, I'm not talking about customer support).

But most of the software is indeed getting worse right now. The industry is still figuring out how to wield these new tools and what to build with them. Those that figure it out faster than others will make billions.

The problem is that in order to win you need to be in the top 1-10%, since you are competing against the same engineers. Being slightly above the average is only "good enough" to not get fired.

And the bar is already raised because every single one of those engineers now has access to the same models. So you need to be in the top 1% of the engineers armed with LLMs. You can't just rely on LLMs getting better. AI is not the great equalizer, it is, in fact the opposite.

> I believe many of the optimizations are premature

Does it work both ways? How hard should we fight premature optimizations that LLMs are introducing? Or the crazy amount useless tests?

If AI does something good, you take responsibility. If it does something bad - "the agent messed up". /s

I wouldn't be surprised if a good portion of these PRs are multiple agents stuck in a loop constantly overwriting each other changes, with no human involvement at all.

Previously there were two problems with NIH solutions:

- they have implementation cost.

- they are usually born from arrogance mixed with lack of understanding of existing solutions and their tradeoffs, and typically end up being a net negative.

Before LLMs, the cost is what primarily used to stop us from building them, it used to be a natural filter for bad ideas. Now that the cost is gone, it's nearly impossible for most teams to stay disciplined and not be building those.

Why wouldn't you build a software factory? If you succeed, the ROI is infinite. Who cares about immediate customer needs, we're building a utopia here. Once it's done, all customer needs will be solved. Just one more month, I promise.

> decisions made by SaaS providers that don't work for you

Unfortunately this starts to break when there's more than one you on the team.

I see a lot of hand-rolled NIH solutions that create value for one person only, and are a net negative on the team.

> does this advice correspond to our policy docs

This is not a “system one” question.

You’d be much better off using a proper LLM here: more accurate, can provide justification, can be steered when mistakes are made. Jev is going to be a coin flipper with almost no knobs.

> give it a list of probability for each number and ask it whats the probability of each number

Did i hear that correctly? In order for Jev to be accurate you have to give it the answer before asking for the answer?

(btw this is exactly how Jev is playing games).

Ha, two years is too generous.

If they weren't lying about $100M ARR, they had to be at 6.5T/day for a week to hit that number.

They also claimed they were at 1T/day just couple weeks ago. That's $15M ARR.

85% drop in just a couple of weeks?

These are very hard problems to solve.

It's often nearly impossible to know in advance how hard the task will be, so the model has to guess, and will inevitably miss. Just because you know how hard the task is, doesn't mean it's obvious. Often it's only obvious to you, because you have additional context that models are missing.

A bad guess can lead to either context rot (if subagent wasn't used), or slower execution (if subagent was used but it turned out to be a bad idea). Both are bad UX.

Now, we could improve accuracy by having the harness do a research on each task before executing... which leads to even slower execution, also bad UX.

At this point there's no need to pay a data scientist. You can literally ask Claude to do everything for you: extract real examples, classify them, post-train a model, and ship an API.

Now, it could be viable if your business has literally hundreds of problems thats require classification. I just haven't seen those.

I treat the fact that almost noone was doing that as evidence that decision models aren't that useful/groundbreaking. That, and the fact that every single demo I saw was either fake (e.g. playing games), contrived, or plain wrong (e.g. using Jev for compaction).

Measuring ARR based on 7 days of data is beyond stupid.

You can create a company with 2B shares and sell one share to your friend for $1000. Lo and behold, you own $2 trillion dollar company, leaving Elon behind.

Jev API returns both confidence and probabilities.

But confidence value is just a function applied to probabilities. It is not coming from the model, and it carries no additional information.

It is documented btw, and yet you will see plenty of claims that Jev is better than LLM because it returns both.

> work on one specific problem, right? That is the opposite of "general purpose".

A business doesn't need Jev for the sake of Jev. Most business are solving specific problems.

And fine-tuning got a lot cheaper these days - I've seen claims here on HN that ~500 examples is enough to beat Jev.

> "nobody thought it's a good tradeoff", which again is false, there was huge demand for a cheap and accurate general purpose classifier

There wasn't. The hope is that there was a latent demand, but we've yet to see if it's truly latent or just manufactured.

Noone is saying "hell yeah, finally we got a general purpose classifier, my business needed it so much". The typical message is "this seems cool, let me see where I can apply it".

The fact that name itself is a play on Jevons Paradox illustrates that there was no demand until Jev was released.

Yeah, but I wouldn't be surprised OpenAI's decision API is a post-trained Luna with confidence calibration.

Typesafe claims that Jev is calibrated, but there are plenty of examples where it completely fails (predicting die roll being the most obvious one).

Unfortunately calibration is hard to benchmark.

> people that came up with a product people wanted

We've yet to see whether this is true, or is it just manufactured demand. There are dozens of Jev demos, but pretty much all of them are either cool but useless, or simply fake (i.e. harness doing 99% of the work).

> Models we had before Jev had at most 2/3 of above qualities, but none of them were 3/3.

You're the one being deceptive here. Jev is trading accuracy, speed, and cost for generality. It's less accurate, slower and more expensive than trained classifiers. So it's still 2 out of 3, but with decimals. Maybe 2.2 out of 3 if I'm being charitable.

And the reason we didn't have that before is because nobody thought it's a good tradeoff.

And what do you think Jev confidence score is?

Here's a hint: confidence is not generated by a model.

> zero hallucination

Plenty has been said about this claim. If you're still falling for this, I feel sorry for you.

If you remove wheels from your car, your car will get a "no speeding ticket" property, and yet there's nothing exciting about it.

> You can’t get that with a fine tuned LLM

Of course you can. All these claims are nothing but marketing.

> I strongly doubt that a copycat that was put together within days after Jev’s release will be able to match it on a sun of its properties

But why? If the simplest way to achieve Jev's capabilities (accuracy, cost, latency) is by fine tuning a small model, what makes you think that this isn't exactly what Typesafe did?

And even if they did something different - what makes you think it was a good idea in the first place, given how easy their results were replicated without any "secret sauce"?

Yeah I have similar struggles: I witnessed several multi-month projects that should never have been started. But they would sound cool on paper, and sycophantic LLMs would support any shit-on-a-stick, so they were kicked off, only to be killed months later. Huge boosts in productivity coming from coding agents are often negated by AI psychosis. But is some ways I enjoy it, because instead of being the great equalizer AI turns out to be the great differentiator.

Review tool is an interesting one - I find LLMs to be quite useful for code reviews, but people should really stop associating code reviews with Github/Gitlab and running them as a part of CI. Those platforms were built for humans because reviewing patches over email was a terrible experience. Clankers have no problem with raw patches, in fact they prefer them over fancy web UI. Code hosting platforms are not a good place for agentic code reviews, and they never will be.

Most LLM reviews should only run locally on developer's machine, in your regular coding harness.

YouTube and even Google Maps were acquired, not built by Google.

Google Video was a competitor to YouTube and it's a perfect illustration of what native Google products look like.

One funny observation is that while LLMs are great at coding, but they absolutely suck at generating content for themselves (or other LLMs).

They are truly horrible at generating docs, prompts, knowledge bases, memories, etc.

Refusing to commit LLM-generated docs (or tests for that matter) requires discipline that most engineers don't have:

- it feels productive and has illusion of value

- it seems harmless/free, refusing means that at best you'll be at net zero

- if you do refuse, now you actually need to think about what should be documented

So yeah, LLM-generated garbage will inevitably spread.

The good news is that teams that do have the discipline are way more competitive today.

Primary use case seems to be posting on social media about Jev.

But AI output could be the very same spreadsheet.

I have a bridge to sell you.