Hacker News Reader

Comment by datadrivenangel | Hacker News Reader

"Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores. For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI."

So AI makes consultants faster but worse. As a consultant, this should make my expertise even more valuable.

View on HN

Seriously, you cherry picked the paragraph you use to cite your opinion, which contrary to the finding of the article itself and completely ignore the the first part of the second sentence of your paragraph "for a task selected outside the frontier" - so, something beyond the capabilities of AI at this time ONLY decreased their correct solutions 19 percentage points.

43% better it made below average people, 17% better it made above average people.

Same paragraph.

BCG consultants. Not average people.

You know, I missed that part. Oops.

Looking into the full paper, it looks like the task identified inside the frontier is much more broken down, with 18 numbered steps. The outside the frontier task is two bullet points.

So for clearly scoped tasks, ChatGPT is hands down good. For less well scoped work it reduces quality.

Interestingly, the authors also looked at the amount of text retained from ChatGPT, and found that people who received ChatGPT training ended up retaining a larger portion of the ChatGPT output in their final answers and performed better in the quality assessment.

' As a consultant, this should make my expertise even more valuable. '

Consultant logic at its finest.

More garbage outputs being produced faster, paired with expertise which makes it credibly sound like more fast garbage will solve any remaining issues, probably _is_ approaching peak consultant value.

Only if that's what the clients want or need to justify actual improvements!

Usually the goal is to produce less garbage outputs slower.

The goal isn’t to generate billable hours?

We like to find win-win situations.

Only if being wrong actually matters.

The research was partially conducted by Boston Consulting Group, so in this case being wrong doesn't really matter as long as the Partners are smooth and the associates create billable hours.

> So AI makes consultants faster but worse.

That was just for tasks not suited to GPT-4 (and I assume, LLMs in general).

> "For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI."

For tasks suited to GPT-4, consultants produced 40% higher quality results by leveraging GPT-4.

My take on that is that people who have at least some understanding of how LLMs work will be at a significant advantage, while people who think LLMs "think" will misuse LLMs and get predictably worse results.

> while people who think LLMs "think" will misuse LLMs and get predictably worse results.

I've found that now that I see LLM's less as thinking and more as stochiastic parrot I get a lot less use from them because I give 3-5 turns at most versus 10 turns.

Well, maybe a balance between a stoic parrot and a "thinking machine" might be the best approach.

You know, everything in moderation and what not.

Yes, ideally.

The problem for me is it used to be an addition to my energy and confidence, but when you "take the magic out of it" it doesn't help out there so much.

In antirez's blog he mentioned how it made things "not worth it, worth it" and I agreed, but that happens much less often if you think an LLM helping you is a fluke.

Reply on news.ycombinator.com