Comment by Volundr | Hacker News Reader

Comment by Volundr | Hacker News Reader

Parent Full thread

This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting.

> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).

Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.

Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".

Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify (I mean that is the nature of them lol).

With enough budget this seems a rather reasonable and fun task.

I'm not sure I agree with the extremely easy to verify part. How do you validate that your new test suite covers the exact same edge cases as your old?

But yes, LLMs are great at tests. I'm not doubting that an LLM migrated some legacy tests, or that it was a task taking a long time. I am doubting the way it was presented in the article, that this was taking a full team of engineers dedicated to nothing but this, 5 years to accomplish (and presumably already spent 4 million working on this, since the total estimate was 6 million, and they've been at it for 4 year).

I think some PR flack got ahold of the fact that an LLM wrapped up migrating some legacy tests that a team had been slowly chipping away at for 4 years, between their normal feature work, and were on track to finish in 5, then wrote it up like it was that teams entire focus instead of a piece of tech debt.

That makes far more sense to me than spending millions for a team of engineers dedicated to nothing but rewriting an existing test suite.

> Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify

I don't think that's true. All you see is all the test pass. You don't know that the tests still cover everything that they used to.

LLMs excel where there is an excellent test suite, and you ask them to modify the thing that the test suite tests, and forbid them from changing the tests.

> I can't see the code before, I can't see the code after. I can't verify they do the same thing.

The ultimate bad faith interpretation. "Unless I can verify the results that contradict my worldview, I don't acknowledge them."

> https://www.mikekasberg.com/blog/2026/08/19/hacking-with-cla...

"I haven't done this, so this doesn't prove it."

> https://www.bbc.com/news/articles/clyq011414eo

"I haven't seen the paper trail, so this doesn't prove it."

on and on...

The comment I was replying to said

> But you _can_ verify if you had bothered

I think pointing out I can't is indeed fair.

How is it bad faith to question corporate marketing blog posts? That's not bad faith, that's table stakes for critical thinking.

We absolutely should be skeptical when an AI company makes big claims. The fact that the company the AI company is talking about also claims the same thing doesn't change that. OpenAI is getting awareness and marketing out of this, and I'm sure Asana is getting something out of it too.

And I'm not even saying OpenAI or Asana are necessarily lying. Asana might not find out for months or years that some tests had been rewritten poorly, and don't sufficiently test the thing they were supposed to test anymore. For example. If they truly had 5 years of work, then I find it hard to believe that in two weeks of the LLM churning, they had the time to review all the new tests. They spot-checked, at best.

Maybe everything is great. Maybe the LLM did a wonderful job, and this was awesome for Asana. But we have no idea, and we're unlikely to ever find out. Unless, of course, it's in Asana's interest from a marketing perspective to tell us.

(Not sure what the URLs you posted in your comment are supposed to prove. They're unrelated to the issue at hand.)

Reply on news.ycombinator.com