Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify (I mean that is the nature of them lol).
With enough budget this seems a rather reasonable and fun task.
With enough budget this seems a rather reasonable and fun task.
But yes, LLMs are great at tests. I'm not doubting that an LLM migrated some legacy tests, or that it was a task taking a long time. I am doubting the way it was presented in the article, that this was taking a full team of engineers dedicated to nothing but this, 5 years to accomplish (and presumably already spent 4 million working on this, since the total estimate was 6 million, and they've been at it for 4 year).
I think some PR flack got ahold of the fact that an LLM wrapped up migrating some legacy tests that a team had been slowly chipping away at for 4 years, between their normal feature work, and were on track to finish in 5, then wrote it up like it was that teams entire focus instead of a piece of tech debt.
That makes far more sense to me than spending millions for a team of engineers dedicated to nothing but rewriting an existing test suite.
I don't think that's true. All you see is all the test pass. You don't know that the tests still cover everything that they used to.
LLMs excel where there is an excellent test suite, and you ask them to modify the thing that the test suite tests, and forbid them from changing the tests.