hyperpape

> The curve of AI improvement will continue at the current pace

I guess this is trivially true if you say "maximalism" (hell, the maximalists think it will speed up as the AI becomes a super-AI-researcher), but as long as the rate of change is positive and not miniscule, it's hard to predict what 2035 looks like in software development.

These things are very hard to quantify, but making the progress that happened from Jan 2025-December 2025 repeat twice in 10 years would be enough for me to say I couldn't predict the day-to-day of a software engineer in 2035.

We need a companion to "IN MICE", which is "IN EVALS".

I don't think this is bad research, but you have to understand how far it generalizes. I'm not saying that evals are useless, we need to do our best to produce good benchmarks. But benchmarks are always going to lag pretty far behind real world applications.

> According to the paper, “Of the 22 vulnerabilities, five were level-based, meaning that the default weak isolation level led to the anomalies behind the vulnerabilities. The remaining 17 were scope-based, meaning that the database accesses were not properly encapsulated in transactions and concurrent API requests could trigger the vulnerability independent of the level of isolation provided by the database backend.”

I don't want to commit to a real opinion, but the cynic in me sees a bitter lesson you could take from this is that the database should default to a low isolation level--the damn developers aren't even using transactions right, so why waste performance handling transactions in the strictest possible way?

I wonder what the impact of the rising base rate of employees with college degrees is. In 1992, a fresh college graduate had better educational attainment than 42% of the labor force. In 2016 (latest date I found numbers for), that was down to 32%. https://www.bls.gov/spotlight/2017/educational-attainment-of...

That shifting distribution would somewhat reduce the advantage of a college degree against the average member of the labor force.

> If AI tokens were so magical in creating new value in developing software applications generally, they wouldn't be selling tokens directly. They'd hoard the tokens are use them to dominate SaaS software in any industry they want.

This doesn't follow at all. Anthropic's revenue is growing 10x year over year selling tokens. Their tokens can be super magical, let them enter established industries and displace incumbents, and get 100% annual growth in those industries, and they would still be better off prioritizing selling tokens, because it's a great business.

What your argument shows is that there are limits. Their tokens are not quite powerful enough to make infinite money instantly in every area of software. Admittedly, that does seem true.

Management is very prone to fads. The current fad is that middle management is useless. Tomorrow, they'll discover the idea that organizations can have employees "working hard" on things that no one cares about, and that someone actually needs to work on focusing that effort.

Of course, the truth is you can have too many middle managers or too few (it really was bad that in 2017, the biggest achievement was "growing headcount"). But fads have a tendency to overcorrect.

Even from the most purely instrumental perspective, what we care about is our ability to make use of correct answers, which is quite distinct from the possession of correct answers.

There are many theorems that aren't directly interesting, but whose proof requires techniques that are of substantial further interest, that lead to new domains, and/or new practical applications. Simply being handed a proof for those theorems isn't enough--we require the ability to apply those techniques in the real world, or discover further areas of mathematical research that build on that proof or its techniques.

It may be that AI can build on its own work for the long-term, but so far, AI does best at exploration in areas that have precisely specified and measurable goals. Actually creating understanding, and making use of mathemtical results outside of pure mathematics is more challenging than simply creating proofs.

I think the field will figure out how to make use of AI, and it will be better off for it. But that is not the same as just saying "answers good, grog want more answers."

Admittedly, that's right, and makes sense for that use case. But as others have pointed out, killing the user's web browser while they're using it is equally painful.

But the second clause doesn't follow from the first!

I don't think Linux was plausibly going to remove the OOM killer in 2004 or later. So the right solution for Linux is very much to tweak it to be less painful.

I confess, this is very funny and the underlying situation is a bit absurd, but it's unclear what point Brouwer is making by pointing out the absurdity.

There surely is something absurd about having to register specific processes as exempt from the OOM killer. But given that the OOM killer exists, and could kill xlock...how should that be fixed?

> Has there been ongoing, persistent attacks by AI on domain expertise where we can say the moat holds, economically speaking? So far it seems quite the opposite.

What do you mean by this? Most human white collar workers still have their jobs. I can't see the future, but yes, so far, human expertise is doing ok.

We'll see what happens in 2027, and 2028, and...

Like so many posts that end up on HN, I just want to say "you've got a decent idea, but tone it the fuck down."

It's absolutely true that domain knowledge is incredibly useful, and developers aren't always great at gaining it. But there's also something about decomposing systems into their component parts, understanding algorithms, and knowing how code works that's also incredibly useful, even with agents in the picture. A really good developer needs both of those skills.

Take that example, of the generated shift that's illegal (by coincidence, I do freight optimization and work with examples like that in my day job). A domain expert will know the specific example is illegal. So they'll tell the agent to fix it. The agent will probably fix it for that case.

How does the domain expert then know that the agent has produced a thorough fix, as opposed to just that scenario? Not because the agent says so. So it is because they test it manually (but which cases)? Or because they review the strategy of the agent's tests, and know how the algorithms work, and know the edge cases that the tests need to cover? But they can't do that, by stipulation, because they're not experienced with code, they're just using the agent.

So yes, if the agent gets to the point where it can design robust software that avoids edge cases in a complex domain, doing complex operations and is thoroughly tested, and so on, then half of my skills are going to be irrelevant.

Out of the box, agents don't do that today. Perhaps they'll get to that point, but until then, your knowledge of where to put a semicolon has become less useful, but your ability to specify and test processes precisely has not.

But yeah, knowing your domain well is a damn good idea.

They will release a system card, and you can then confirm or disconfirm your assumptions.

No one has to provide a definition to argue that your definition is inadequate.

"Most purely cognitive labor is automatable"

I cannot express how annoyed I am a researcher could use such a shitty definition.

It only makes sense to say "most" if you have a clear idea of what constitutes the majority. "Most people are male" yeah, fine..50% + epsilon of humans are males. That's more or less decidable (maybe a little vague because of intersex folks). I believe it's false because there are slightly more females but it's obviously measurable.

Now, most cognitive labor...what does that mean? Is it most of the time? Most of the tasks? Most of the value? Most of the job descriptions?

If I am a developer, and the majority of my code is written by AI, but I'm still in the driver's seat, is that most of my cognitive labor? Probably not. Ok, what if my company fires 60% of its developers, does that mean most development cognitive labor is automated? Well, it's most of the expense, and most of the butt in chair time, and it's most of the individual jobs, but it's not most of the job descriptions.

Of course, there's no way that all these researchers making pronouncements are giving consistent answers to what they mean by "most". They're probably not using his phrasing either.

Edit: The four options I threw out above: time, tasks, value, job descriptions are each interesting in their own way. My point is not that they're bad questions to be asking, it's that they're all separate questions that matter in different ways.

From my perspective, and the perspective of most academics[0], it is their contribution to human knowledge, which is kept locked up by predatory publishers.

A majority of academics will simply and without hesitation, offer their students and collaborators pirated versions of their own work, because they value knowledge.

Commercial authors may feel differently.

[0] I'm a former Ph.D. student, but my attitude was the same both within and outside of the academic world.

Heart health isn't the only mechanism by which exercise could affect mortality.

Concretely, it has to decide whether it is in a circumstance where that skill is useful, pull the instructions into the context and follow them.

> we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors), if we already have this assumption for software engineers, we should have it for this field as well,

This is a pretty wild leap. Code has a lot of hooks for training via hill-climbing during post-training. During post-training, you can literally set up arbitrary scenarios and give the bot more or less real feedback (actual programs, actual tests, actual compiler errors).

It's not impossible we'll get a training regime that does the "same thing" for medicine that we're doing for code, but I don't know that we've envisioned what it looks like.

I very specifically said I read 25 pages of it in the first post of this thread. I didn’t go back, I haven’t looked at the paper since yesterday.

I read their methods and their explanation and judged them to be lacking.

The fact is, they did not measure that LLMs prefer LLM authored resumes, but that is what their paper stated.

They measured that LLMs prefer LLM authored executive summaries, which is a weaker claim.

Thank you, that's correct.

To be perfectly clear, I understand their justification for only _editing_ the executive summary, it is arguably reasonable, because editing the work history would risk altering the details in ways that compromise the measurement. This is a hard problem to solve (you might try reviewing the resumes for hallucinations, but I can't think of a precise study design that doesn't risk problems).

What is, imho, impossible to defend, is having the LLM only evaluate the executive summary in isolation, and reporting that as it preferring resumes it wrote.

What you've shown is that LLMs prefer executive summaries they wrote. But the overall impact on how they will evaluate your entire resume is not measured by this technique.

Worse, this isn't just "decent paper, bad summary", their abstract misreports their findings.

I'll copy what I wrote on LinkedIn (note: I read roughly 25 pages, which is half the paper, and read it quickly)[0]:

"If I read the paper correctly, they don’t actually show that LLMs prefer resumes they generate.

Their actual method seems to be taking a human written resume, deleting the executive summary, having an LLM rewrite the executive summary based on the rest of the resume and then having another LLM rate the executive summary without the rest of the resume.

That’s likely to massively overstate any real impact, if you can even rely on it capturing a real effect.

I really wonder if I read that correctly, because I can’t come up with a justification for that study design."

[0] I couldn't help but mildly copy-edit before pasting here.

Edit: yes, the authors present a reason for their design, and an ideal version of my comment would've said that. I do not consider it much of a justification. See below: https://news.ycombinator.com/item?id=47987256#47987727.

I do not think it is easy, like I said. I am saying other people are acting like it’s easy.

See the second comment on this article. https://news.ycombinator.com/item?id=47976781

See @emp17344 responding to me.

I love how these articles drop, and all of a sudden HN is filled with people who think engineering productivity is simple to measure.

Yes, productivity implies revenue (or cost reduction), and revenue is measurable.

However:

1. You spend money today to build features that drive revenue in the future, so when expenses go up rapidly today, you don’t yet have the revenue to measure.

2. It’s inherently a counterfactual consideration: you have these features completed today, using AI. You’re profitable/unprofitable. So AI is productive/unproductive, right? No. You have to estimate what you would’ve gotten done without AI, and how much revenue you would’ve had then.

3. Business is often a Red Queen’s race. If you don’t make improvements, it’s often the case that you’ll lose revenue, as competitors take advantage.

4. Most likely, AI use is a mixture of working on things that matter and people throwing shit against the wall “because it’s easy now.” Actually measuring the potential productivity improvements means figuring out how to keep the first category and avoid the second.

This isn’t me arguing for or against AI. It’s just me telling you not to be lazy and say “if it were productive you’d be able to measure it.”

I think this site is doing a binary search, so that you narrow down on a boundary.

It would be much funnier, and also more insightful, if it didn't do this and let you contradict yourself.

Unless kids have gotten a lot faster in the past 25 years, I think that's a lot better than a typical 2000 person high school.

> “The raw output of ChatGPT’s proof was actually quite poor. So it required an expert to kind of sift through and actually understand what it was trying to say,” Lichtman says. But now he and Tao have shortened the proof so that it better distills the LLM’s key insight.

Interestingly, it was an elegant technique, but the proof still required a lot of work.

Which is difficult, because the fact that you can come up with your example questions tells us they're probably not very dangerous. Plenty of ink has been spilled about how LLMs could help people create bioweapons. The basic idea "you could do dangerous things with an LLM" is already pop culture, and you're not doing anything dangerous by giving easy example questions.

A dangerous question would have to be along the lines of "Could I use unobtanium with the Tony Stark process to produce explosives much more powerful than nuclear weapons?" so that the question itself contains some insight that gets you closer to doing something dangerous.

Perhaps the reason for not publishing the questions is twofold: 1) they want a universal jailbreak that can get the model to answer any "bad" question. 2) they don't want bad publicity when someone not under NDA jailbreaks their model and answers their question

That's a real question, maybe the changes are useful, though I think I'd like to see some examples. I do not trust cognitive complexity metrics, but it is a little interesting that the changes seem to reliably increase cognitive complexity.

Why are you replying as if I denied second hand smoke harms people? I very clearly said it did.